Why Ministries Aren’t Improving: Outcomes Measurement in a “Wicked” World

Listen to this article:

In the past century, goods and services offered by businesses have improved dramatically. In every industry — finance, transportation, retail, etc. — quality and affordability have increased. Sadly, that’s far less true in the nonprofit world. Despite advances in supporting technology, case management and soup kitchens aren’t producing dramatically better results than our great-grandparents saw.

Why is that the case? 

It’s not a lack of funding, which has increased on a per capita basis several times faster than inflation. Instead, the answer lies in understanding “information environments” — which refer to the natural systems of information and feedback we use to make decisions.

Some are what behavioral scientists call “kind” environments. They are characterized by immediate, accurate feedback and consistent information flow, analogous to using a GPS to navigate a city with clearly marked roads: It’s easy to see where to turn next; and if you take a wrong turn, your GPS will recalculate and get you back on track. For instance, astronomy operates in a relatively “kind” environment. The movement of stars and planets follow predictable patterns, quantifiable rules, and offer crystal clear feedback on predictions.  Consequently, astronomical events like eclipses can be forecasted to the second and astronomers’ opinions tend to converge over time.

Other information environments are like navigating a trackless desert with no roads, signs, or cell signals. It’s easy to veer off course and never know it. This is called a “wicked” environment and is characterized by delayed or inaccurate feedback and inconsistent patterns of information flow. For instance, political pundits prognosticating about a new government policy can be influenced by extraneous factors that make it easy to post-rationalize any outcome as a success (or failure). This “wicked” environment explains why pundits have not converged in their views like astronomers have. 

If we apply this framework to business and nonprofits, we see that most businesses receive rapid and accurate customer feedback. If it’s ignored, someone else will take their customers. This drives the industry to produce better products and services over time — even if a given business isn’t operated effectively.

Nonprofits have no such auto-correcting features. They can stay afloat for decades by engaging donors with occasional success stories or feel-good volunteer experiences, ignoring whether charity recipients truly benefit. And since handouts will often be received with gratitude, even client feedback surveys may be insufficient to redirect a nonprofit to address root causes. This means many ministries are fully insulated from information that could help them improve their programs.

The good news is there are ways to mitigate this difficulty. Most of the medical field operates in a “wicked” information environment. When the town doctor of old listened to your heart and prescribed a bottle of pills, there was often no way of knowing whether his diagnosis was correct. Since patients often didn’t come back after treatment, he might not know if the patient improved or declined. Even if he knew his patient’s condition over the weeks and years following treatment, he would have never known for sure what role the medicine had on the outcome.

In that kind of information environment, medical advances were slow and sporadic, and widely adopted “innovations” were as likely to harm as help (e.g., mercury treatments and bloodletting). Yet, the field has steadily improved its patient results in recent decades. 

They accomplished this by combining the scientific method (to determine general best practices) with patient outcome tracking (to improve care at the local level). Think of it as using a map and compass to navigate through a barren landscape. It’s still a challenging information environment but it can be overcome with careful effort and the right tools.

This model of researched best practices combined with localized outcomes measurement can enable nonprofits to overcome challenges they face. They can proxy scientific research by learning from other organizations with exemplary results — and access well-designed experiments, too.

But even if programs are based on what works elsewhere, there’s no substitute for testing their local effectiveness. That involves more than measuring how much is given away. Measuring long-term results like stable housing, employment, education, and family reunifications should be prioritized. With the right tools and program design, it’s even possible to measure knowledge, motivation, and social connectedness. 

A common objection is that fleeting interactions with numerous clients makes measuring long-term effectiveness nearly impossible. Unfortunately, the same superficial interventions that make outcomes hard to measure make it likely a program isn’t producing much worth measuring.

The good news is that while effective measurements are a necessary cost of doing good work, they also help fund it. Donors (particularly major ones), granting organizations, and volunteers  have entrusted the organization with their gifts. They deserve to know their generosity makes an impact. When they see positive results, they frequently increase their contributions. Even paid staff will be encouraged by demonstrable results that affirm their sacrifice and service. 

Furthermore, effective outcomes measurement supports clients’ personal growth. Data collection usually takes the form of goal-setting conversations which enable clients to see positive change over time. For many, it’s the first time in years (if ever) the trajectory of their lives has been positive. Improving numbers on things like credit scores or weeks of sobriety gives them confidence their progress is real. It’s very true that in life, as in sports, people play differently when they keep score. Tracking progress and goals will inspire them to do their best.

There’s a final reason to measure outcomes: If we don’t, we may waste much of our career. None of us wants to realize that after decades of laboring in love, we are feeding the grandchildren of our original clients. But we are not doomed to wander in the wilderness. A combination of general best practices and local measurement can guide us and the people we serve to a better future.

Take your impact to the next level with True Charity’s Outcomes Toolkit, designed to equip you with actionable insights and proven strategies for effective charity.

And discover all the ways the network can help you learn, connect, and lead change here.
Already a member? Unlock your full range of benefits through the member portal.