We cannot march through the 21st century with the administrative systems of the 19th century," Prime Minister Narendra Modi announced in a recent speech delivered at the government think-tank Niti Aayog. The quip served as a welcome acknowledgment that the essential bureaucratic organs of the Indian state are badly out of sync with today’s demands.

The state of the Indian Administrative Service (IAS), the country’s elite civil service cadre, confirms this fact. Of the 3.3 million individuals employed by the public sector, the IAS constitutes a tiny fraction—totaling fewer than 5,000 officers. Yet, because it occupies the very nerve centre of the Indian state, no single bureaucratic entity receives as much scrutiny. While a competent, functional IAS may not be a sufficient condition for improving governance outcomes, it is likely a necessary one.

The charges levied against the bureaucracy, and the IAS more specifically, run the gamut; it is hamstrung by political meddling, governed by outmoded personnel procedures, crippled by a lack of domain expertise, and marked by a mixed record on policy implementation. It is no surprise then that there are opinion columns penned on a daily basis calling for everything from the complete dismantling of the IAS to modest tweaks of its promotion regulations.

Although there is no shortage of opinions related to the IAS, there has been a surprising paucity of hard data on its operations and performance until now. A new body of research, combining unprecedented access to the profiles of IAS officers with granular data on local development outcomes and electoral dynamics, sheds new light on their career trajectories, their impact on development outcomes, and their relationship to politics.

For officers early in their careers, Union Public Service Commission entrance exam scores and demonstrated improvement during the post-entry training period are positively correlated with officers’ perceived effectiveness, as judged by a diverse set of stakeholders, including civil society leaders and politicians. But, while initial characteristics shape career trajectories, they are not deterministic. In the long term, there are rewards for officers who invest in training and specialized skills.

For the first time, we have highly disaggregated data which demonstrates that individual bureaucrats can have strong, direct, and measurable impacts on tangible health, education, and poverty outcomes. But, there is considerable variation on this score. Older officers who enter as part of large cadres and face limited career prospects—given the fixed retirement age—tend to fare worse at improving economic outcomes. In a surprising twist to conventional wisdom, some evidence suggests that officers serving in their home-state are linked to superior service delivery. While local bureaucrats are typically thought to be susceptible to corruption, the presence of strong local accountability mechanisms, such as a strong media presence, can act as a check on malfeasance.

However, political interference poses a constant threat to bureaucratic functioning. Historical data suggests there is a 53% chance that an IAS officer is transferred in any given year. As a result, political loyalty rather than professional qualifications often represents an alternative path to success. However, politics can also work in surprising ways. Where elections are less competitive, some bureaucrats actually appear better motivated to do their job, presumably because there is less uncertainty about who will wield political power in the future. This finding is at odds with the prevailing belief that greater electoral competition incentivizes better bureaucratic performance.

This new literature suggests several areas for reform.

First, there is valuable information that can predict the future effectiveness of civil servants, yet those in charge of making personnel moves rarely utilize these data points. The advent of big data provides a natural opportunity to use metrics on officers’ performance in the field to inform promotion and retention decisions. Seniority, after all, is a blunt instrument for deciding who gets promoted and who does not.

Second, given that older officers entering the bureaucracy are perceived as less effective, reducing the maximum age of entry into the IAS is a relatively easy reform the government could introduce. A much thornier issue is tackling under-performing officers already in the service. Here, the government should consider the proposal that officers deemed unfit for further service at specified career benchmarks be compulsorily retired through a transparent, uniform system of performance review.

Third, the government might contemplate allowing IAS officers to work more closely with their home states. Although India’s founders chafed at the prospect that officers be too closely linked with their state of origin for fear of elite capture, this issue could be revisited for further consideration. There is room for experimentation on this front.

Finally, it is imperative that the Central and state governments institute safeguards to protect against arbitrary, politically motivated transfers and postings of civil servants. Despite judicial prodding, most states have stalled on such moves.

Modi ended his Niti Aayog remarks by declaring that “a transformation of governance cannot happen without a transformation in mindset. A transformation in mindset cannot happen without transformative ideas." While far from revolutionary, the policy changes suggested by new research on the Indian bureaucracy promise concrete benefits. As current chief economic adviser Arvind Subramanian noted in 2012, there is a race between rot and regeneration in the underlying institutions of the Indian state, and it is far from clear that the latter is winning.

Saksham Khosla and Milan Vaishnav are, respectively, research analyst with Carnegie India and senior associate with the Carnegie Endowment for International Peace

This column is based on a new study, The Indian Administrative Service Meets Big Data

Comments are welcome at theirview@livemint.com