By Sudha Ramen
During my tenure at the State Planning Commission, I had the opportunity to lead and coordinate several impact assessments and evaluation studies of government schemes. That experience gave me a clear perspective and an interest in understanding this discipline. I could see, on one side, the enormous volume of information that the government generates every single day, and on the other side, the difficulty that decision makers face in converting it into consolidated evidence or understanding its outcomes. The actual gap here was not the data, but the systems, skills, and culture needed to use this data effectively. When I joined the Government Analytics Virtual Program offered by the World Bank, I found that there is a potential solution to this gap. Government analytics is now becoming a global discipline, with tested frameworks and evidence from governments across the world. This essay is an attempt to connect the government analytics discipline and the existing opportunity for the governments, drawn analytically and practically from my experience, to address the challenges that governments face today and how the power of data can be a potential solution to overcome them.
Data as the new currency of the AI era
In the digital economy, data is popularly described as the new currency. Like currency, data stores value, and that value grows when it circulates. A record that sits unused in a department ledger is like money left unutilized. The same record, when joined with other records and analysed, can reveal where public finance could be adequately allocated, avoiding leaks if any, which schemes perform better, and where the regional focus must be. In the era of artificial intelligence, the value of the data has multiplied, because AI systems can now read millions of records, detect patterns, and generate projections at a speed and scale that no human team could match. The private sector understood this early, and the world’s top corporates have harvested the benefits of this data by monetising it. The scale of the public treasury is easy to underestimate. Public procurement alone accounts for roughly twelve percent of global GDP, which means that a government analysing its own procurement records is analysing one of the largest financial flows on earth.
Governments hold a far larger treasury of data than any corporation, yet most of it lies dormant. Much of what a government does is recorded in some way as a standard part of public process, and these records are valuable in themselves when digitised.2 Public administrative data has two properties that even the largest private datasets rarely match. It is highly granular and quite comprehensive, because it covers the entire population rather than a sample. The digitization of administrative data offers governments the opportunity to strengthen public administration in a way that was considered unimaginable so far. The question this essay examines is why that opportunity remains largely unrealized, and what it takes to realize it in this digital era where most of these records are already digitized.
The challenges governments face today
The first challenge is fragmentation. Government data sits in silos because government departments are built in silos. Each department maintains its own systems, designed for its own record keeping rather than for analysis. Formats are incompatible, definitions differ, geographies do not align, and frequencies vary – some updated daily, monthly, half-yearly or annually, and sometimes only when a survey is concluded. The result is that though departmental data is in full shape, what is needed for a whole-of-government analysis is mostly lacking.
The second challenge is the weakness of measurement. Government analytics describes the public sector as a production function – with inputs, processes, culture, frontline agencies and outputs/outcomes as its core components (Figure 1). Government converts inputs, namely personnel, goods, and capital, through processes, namely management practices, information technology systems, and the culture and behavior of officials, into the work of frontline agencies, which deliver services, collect revenue, and build infrastructure, producing the outputs and outcomes that citizens experience. Every stage of this chain generates data, from payroll and human resource systems, to procurement and budget records, to sectoral case data, to household and citizen surveys. Yet in practice, each stage is measured in isolation, if at all. The result is that some common bottlenecks persist undetected: high staff turnover, demotivated staff, differential prices paid for identical goods, collusion in audits, and outcomes going unmeasured. Each of these deficiencies leaves its mark on the data it generates, but the gaps go unnoticed unless the data is read together – and that rarely happens without a special effort.
The third challenge is the timing and design of evaluation. In most administrations, evaluation arrives at the last stage, sometimes only after a scheme’s period has ended. By the time the report comes, key decisions have already been taken, and the findings change little. The Government Analytics Handbook recounts a telling case from Nigeria. A ministry received 475 million US dollars for water infrastructure, and budgetary releases were made on schedule. Yet when tracking teams visited the sites, the projects were simply not there. Poor design adds to this problem of timing. Many programmes are launched without first fixing clear, quantified outcome targets, so there is no yardstick to measure the scheme against later. And when outcomes do improve, governments rarely verify whether the scheme caused the improvement or it happened for other reasons. Without this clarity, governments risk continuing schemes that never worked and dropping schemes that actually did.
The fourth challenge is the narrowness of the evidence base. Administrative numbers, however comprehensive, carry forward the biases of the processes that generate them. They appear to be perfect figures on paper while the reality on the ground may be different. The principle of a balanced data suite addresses exactly this danger. It holds that no single data source or single type of source should be trusted alone, that quantitative data must be complemented by qualitative evidence which reveals the unobserved factors driving variation in outcomes, and that all measurements must be tied back to a theory of change.
The fifth challenge, underlying all the others, is the scarcity of analytical capacity and culture. The limited use of government analytics has both good and bad reasons, namely the genuine limits of measurement, and the persistence of old practices respectively. Data analysis is still widely treated as a specialist’s task rather than a mandated public sector trait. Analytical questions are still framed around datasets and methods rather than around clearly defined problems addressed by the schemes. In most cases, this task of data analysis is outsourced to a consulting firm or an academic institution without investing in the government staff’s own capacity.
The power of data brings in a solution
Government analytics, as defined by the World Bank, is the practice of leveraging data to improve public administration. It offers a distinct, micro-data-based approach to enhancing state capacity, resting on one foundational idea: measuring the state allows us to manage it better. An administration that measures itself can detect areas for improvement, act on them, and then assess whether its actions worked, improving the public sector both government-wide and one organisation at a time.3 In doing so, analytics serves three purposes – encourages learning within the administration, strengthens internal accountability, and deepens transparency toward citizens – leading to a modern ‘good governance’.
1. To overcome the fragmentation, the practical answer is data engineering and standardization. Pipelines can extract data from departmental systems, clean it, reconcile definitions, align geographies, and harmonise frequencies. This will help to make the records function as a connected analytical asset. This is quite technical work, but it is the backbone. Once the backbone exists, the repurposing of administrative data becomes a systemic practice and will not remain episodic, and the opportunity that digitisation has created will support this backbone.
2. With respect to the weak measurement, the production function itself supplies the map. It tells an administration precisely which data source illuminates which stage of its machinery: payroll and human resource data for personnel, procurement and budget data for goods and capital, public servant surveys and process analysis for management and culture, case and task data for frontline delivery, and household and citizen surveys for outcomes. The international evidence shows what happens when this map is used.
Customs administrations in Madagascar, Tunisia, Kenya, and Uganda analysed their own transaction data to detect tariff evasion and corrupt pairings between brokers and inspectors, and the analysis led to concrete remedial action, demonstrating that rapid and sustained improvements are feasible when data, partnership, and political will come together. Procurement agencies have used spend analysis, interactive dashboards, red-flag systems, and bid-rigging screening tools to generate fiscal savings and strengthen integrity. The gains extend to frontline services with equal force. Chile’s Ministry of Health used administrative data to reduce missed medical appointments among patients with chronic conditions, saving hundreds of millions of dollars, and Guatemala’s Ministry of Education cut the dropout rate of students entering lower secondary school by around nine percent by using its own data to identify children at risk of leaving. None of these findings needed a new dataset. They used existing data, and analytics supplied the insights needed for corrective action.
3. Against delayed evaluation, the answer is to make concurrent evaluation and impact assessment inherent to the system. When a measurement framework and a theory of change are built into a programme right from its inception, administrative data can track outputs continuously, and impact assessments can test outcomes during the programme implementation period itself.
4. To address the narrowness of evidence, the balanced data suite points to instruments that the Indian system already possesses. Third-party evaluations bring independent eyes, free of the incentives that colour internal reporting. Social audits go further and bring the citizen’s feedback into the record. When a social audit finds that a facility exists on paper but not in reality, it provides the qualitative information which otherwise the administrative data can never generate on its own. This also helps to test the assumptions of the theory of change. Such audits provide a strong evidence base.
5. Finally, to tackle the scarcity of capacity, the answer lies in treating analytical skill as core administrative skill. Data analysts and engineers should become a part of the public system, and every officer can be taught the two habits: to start from a clearly defined problem rather than from a dataset or a method, and to identify which data sources matter for the management challenge at hand. Developing this capacity will help the government to reap the benefit of the volume of data it generates and guards.
From data to intelligent systems
Data analytics, coupled with policy analysis, will improve the foresight function of the governments. Data analytics answers two questions: what is happening, and what caused it. Policy analysis answers the next two: what does this mean for our people, and what should the government do now? When these two are clubbed together, it allows a state to build projections of where each sector and region is heading, compare those trajectories against its goals, and construct an informed vision for the decade ahead. This is the analytical foundation of intelligent systems for governance. The system will have permanent arrangements in which data flows continuously from source, artificial intelligence will detect trends and generate projections, gaps between performance and targets get flagged automatically, and decision makers see the whole picture at any given moment to take informed decisions.
India already has working examples of this idea, and they can be read as three generations of the same evolution. The first generation is sectoral. Under the Aspirational Districts and Aspirational Blocks Programmes, data across five development themes, spanning health, education, agriculture, financial inclusion, and basic infrastructure, is collected and analysed monthly in near real time. An equivalent to this at the state level in Tamil Nadu, the Focus Block Development Programme tracks progress through convergence of schemes across seven development sectors using fifty indicators every month, and publishes annual rankings.
The second generation is cross-sectoral and national. NITI Aayog launched the Viksit Bharat Strategy Room in March 2024 as a step in data-driven governance. The room integrates data across states, sectors, and government programmes on interactive screens, and its AI-driven software, built on machine learning and natural language processing, helps policymakers interpret complex data collated from various ministries and national agencies, and monitors national indicators in real time.10 The model has since been extended to the Lal Bahadur Shastri National Academy of Administration, where a similar Strategy Room now familiarizes officer trainees with evidence-based policymaking, embedding the culture of data-driven governance in the next generation of administrators.
The third generation is cross-sectoral and localized, at the level of the state. The Strategy Lab being built under the State Planning Commission is an intelligent system of this generation, designed around the specific realities of a state government. In simple terms, the Strategy Lab brings the scattered data of the government into one place in a presentable format. Its data engineering layer breaks the silos, drawing data from departmental systems and standardising its definitions, geographies, and frequencies. Its analytical layer, connected to AI systems, studies trends across the key development and social sectors, generates projections, and identifies the gaps between current trajectories and the state’s targets. Its policy analysis layer then couples these machine-generated signals with human judgment, field knowledge, and the theory of change behind each programme, converting findings into usable recommendations. The design is aimed at addressing the systemic deficiencies described earlier – the silos, the incompatible formats, the delayed evidence, and unplanned allocation. The practical outcome of such Policy and Strategy labs is expected to be substantial – informed planning, evidence-led deployment of resources, finance allocations tied to need and performance, growth projections built from the ground up, and a steady movement toward decentralized, sustainable growth across all sectors and all regions. One other example of such an intelligent system developed by SPC, using spatial (GIS) and quantitative data emanating from various research studies, is TiNAI.
One distinction is worth making here. There is a growing culture of real-time dashboards and tracker systems in most departments and government agencies today, and this is a welcome trend. But how many of these are built on a proper data backbone, with the data systems institutionalised behind them? Without that foundation, sustaining these dashboards and keeping them truly real time becomes a challenge, and many of them fade after an initial period of enthusiasm. In any case, the subject of this essay is not departmental dashboards or data visualisations. The subject is the integration of these systems into a whole-of-government analytics capability – one that is available to policymakers and high-level authorities, breaks the siloed vision of individual departments, and presents development indicators in a panoramic view.
The economics of such investments could be measured as a return on public investment. Governments spend enormous sums every year on schemes whose performance they discover only years later. An intelligent system built on a strong data backbone costs a small fraction of any major scheme and it could help improve the performance of the schemes, because it improves the quality of every decision.
From utopia to blueprint
It is fair to ask whether all this sounds utopian. A government that uses all its data from records and repository, evaluates its schemes while they are implemented, listens to audit reports and citizens, and plans through intelligent systems may look like an ideal administration beyond reach. But when I look at what such a vision requires, I find that most of it is already available. The data is generated every day as part of routine administration. The methods have been documented and tested by agencies like the World Bank across dozens of countries. The institutions are in place too – planning commissions, evaluation departments, audit bodies, and training academies. Even the early working models can be seen today just like the Viksit Bharat Strategy Room in New Delhi and the Strategy Lab taking shape in Chennai. What is yet to be done – is the harder and slower part of the whole process: assembling all of this together, which needs patience, strong vision and sustained political/administrative will more than any new invention.
Having spent some years on evaluations and now understanding the potential of government analytics to improve governance, I have come to a simple conclusion. In the AI era, data is the new currency, and governments hold more of it than any other institution – though they rarely realise this. A state that measures itself can improve itself. To do that, it should build intelligent systems around those measurements and allocate its resources wisely – that could take growth to every region and every citizen. That is the power of data, and every government already holds this power in its own repository.
About The Author – Sudha Ramen is an IFS officer with 14 years in civil services, Tamil Nadu cadre. A Chevening CRISP Fellow at the University of Oxford & a World Bank Government Analytics Fellow (2026), her career has spanned the field, the state secretariat, & policy fora — beginning at IGNFA Dehradun
Disclaimer– (The views and opinions expressed in this article are solely those of the author and do not necessarily reflect the views of Indian Masterminds. For feedback or queries, please write to [email protected].)
Also Read – A peep inside India’s boldest breeding experiment that is likely to save a species from extinction
