Basically we don’t know. A fair amount of research has been devoted to this topic. But Melinda Tuan’s recent paper for the Gates Foundation – Measuring and/or estimating social value creation – shows that we are very far from any quantitative answer.
Tuan’s overview – which is subtitled Insights Into Eight Integrated Cost Approaches – does not deal specidically with libraries. She addresses the question of impact measurement in general. The difficulties she points to – empirical, analytical and methodological – are generally valid. Whether investments in libraries come from philanthropy (Carnegie, Gates) or from the government does not matter. The same reasoning applies.
Her main conclusions are:
Five summary points regarding the eight integrated cost approaches to measuring and/or estimating social value creation are worth recounting:
- Integrated cost approaches to measuring and/or estimating social value are still in the nascent stages of development due to the lack of maturity in the field of social program evaluation.
- The eight approaches profiled represent a variety of philosophical purposes for blending costs and social outputs, outcomes, or impacts: internal decision making cross portfolios, internal decision-making within portfolios, and general promotion and field building. They also serve varying practical purposes: making prospective investment decisions, informing ongoing practice, and retrospectively evaluating philanthropic investment decisions.
- There is no perfect or precise solution. Each method has its strengths and weaknesses, and no single method has been widely adopted throughout the social sector.
- There are many unresolved technical and big picture issues embedded in the methodologies which determine the distance between the theory and the practice and affect overall utilization.
- The lack of a common language, common measures, quality data, and incentives for transparency represent key limitations for the utilization of any efforts to integrate cost into measuring and/or estimating social value.
Tuan emphasizes the role of high-quality data – and of quality conversations among professionals.
Ultimately, the sector‘s largest efforts will not be about choosing the right model or method. Rather, the most significant effort will involve getting the right data to make whichever model or methodology a foundation or nonprofit organization chooses useful. Without high quality data, any practitioner‘s results will be based on one assumption after another or ―layers and layers of garbage. ….
Lastly, it is important to emphasize that any data, high quality or not; and any model for analyzing data, high fidelity or not, are subject to interpretation. The same data can be interpreted by different people and organizations to reach diametrically opposed conclusions.
The true value of high quality data and analyses of any integrated cost approach to measuring and/or estimating social value creation will be to stimulate high quality conversations about the implications.
Source text: pages 24-25