TOOLKITS Innovation Toolkit for UNDP Signature Solutions
Active Filters:

Using text mining algorithms to analyze public programs performance

What it does.

This tool allows you to use a text mining algorithm to analyze the justifications, given by public officials that oversee the implementation of public programs, regarding the results of the performance indicators of the program, uncover common issues across programs and rank (for based on relevance) the root causes behind the underperformance of government-funded programs.

Value proposition for the government/other partner.

By using a text mining algorithms to analyze public programs performance, you enable the Ministry of Finance to gain a deeper understanding of the root causes behind the underperformance of public spending and aim to make public spending more efficient. This understanding provides crucial insights into the support required by different government branches to enhance the impact of their programs. Such efforts align with UNDP’s governance signature solution and SDG 16, which focuses on building effective, accountable, and inclusive institutions at all levels.

Why and when to use it.

This tool is most effective when governments have already initiated a journey toward results-based budgeting, aiming to enhance the efficiency and effectiveness of public spending and the delivery of services. It’s essential to acknowledge that implementing a results-based budget can be challenging, requiring changes in organizational culture, data collection and analysis capabilities, and reporting systems.

Known issues and troubleshooting.

If there is a lack of transparency, accountability, and aggregation of data, it’s probably not the best time to make use of this tool. In such a case, teams should work towards fostering changes in organizational culture within the government, improving data collection and analysis capabilities, and enhancing reporting systems to ensure that the relevant datasets are available.

Context.

UNDP Mexico and the Performance Evaluation Unit of the Ministry of Finance opened an active line of collaboration to explore ways to improve Mexico’s Performance Evaluation Systems and Results-Based Budgeting—its structure, processes, the information it generates, its usability, and any element that could help make it more relevant to better serve its purpose. The aim of this system is to ensure that the government delivers better public goods and services, improves the quality of expenditure, and promotes accountability and transparency. In this context, the Accelerator Lab Mexico was invited to participate in these discussions. A complex element to be addressed during the process was identified: How does the human dimension affect what people report and the quality of the information generated? Because, although the objectives of the Performance Evaluation Systems are clearly stated in the laws and decrees that support it, those who ultimately interact are people with different motivations and incentives. So, it should not be taken for granted that everything will be perfectly aligned. Instead, the Lab set out to analyze the system and learn from people’s experience what improvements can be proposed to increase the capacity to incorporate learnings at scale.

In the case of Mexico, implementing a results-based budget approach involved establishing a common framework for evaluating public spending performance, extending beyond expenditure tracking to assess the social and environmental impacts of government spending on key dimensions of sustainable development. All programs reliant on public spending are required to define a set of performance indicators and goals set by government officials, establishing a clear causal relationship between program activities and their purpose. This progress is regularly monitored, and the reported information, including text entries justifying why officials believe an indicator did not meet its intended goal, is made available as open data.

For the tool implemented in Mexico, the information is sourced directly from individuals working within the government who oversee the implementation of public programs. This process involves creating a text mining algorithm and feeding it thousands of text entries written by public officials. These officials are asked to provide justifications, in their own words, for why a performance indicator in the program’s logical framework did not reach the established goal for the budget cycle. As individuals on the frontlines of state budget execution, they consistently arrive at conclusions that can be transformed into lessons to enhance public spending. By exploring patterns in this unique dataset of individual explanations, we uncover issues common across all branches of government, aiming to make public spending more efficient.

Cost to implement.

This is a very cost-effective solution that can be funded with less that USD 100.000.

Time

A minimum of six months is needed to implement the solution fully.

People.

You will need to collaborate with the office in charge of the evaluation of public programs in your country. This is typically a unit within the Ministry of Finance. Buy-in from a high-ranking official is a must, and a team of experts within the unit who know the structure of the systems is also necessary (this team must include two people from the public institution’s technology department who will verify the model developed by the consultant). Additionally, you will need a consultant who can write complex Machine Learning (ML) and Natural Language Processing (NLP) model algorithms to help you navigate the code and adjust it to your needs. Additionally, you will need one person to coordinate the project.

Focal point.

Luis Fernando Cervantes

Country, year, and language.

Mexico, 2023, Spanish

Resources.

Signature Solutions
Governance
Enablers
Digitalization
RBLAC Priorities
Effective governance