ASP Paper: “Measuring Success: Are We Winning?”

By Tim Foxley

Summary

Measuring Success, May 2012 This is an important and interesting paper from the American Security Project on the war in Afghanistan.  The paper poses important questions: “What is success?”, “What are the aims of the war?” and criticises the US government for “strategic incoherence at its most stark”.  It bemoans the lack of usable analytical indicators to allow progress in Afghanistan to be judged and suggests nine crucial ways of measuring the political and military situation.

However, overall the paper has flaws and omissions and is a disappointment as a result.  The nine metrics suggested by the ASP are a curious mix and feel very incomplete.  The desirability of having the metrics clashes with the practical reality of getting them and the challenges of how best to use them.  There are other metrics of equal and greater importance .

Background

The American Security Project has produced a paper entitled “Measuring Success: Are We Winning?“, by Joshua Foust.  The paper explains itself thus:

“As we look toward the Chicago Summit and NATO’s coming decisions about Afghanistan it is important to understand how the war is going. And while some things seem worse than ever, we just don’t have a good idea of whether we are progress toward achieving President Obama’s strategy or not.”

“We originally published the metrics paper last October, as the war passed its tenth anniversary. Now, with a major NATO summit approaching this month in Chicago, we’ve updated the report to take account of the last six months… none of which bode well.”

“Afghanistan is going to be one of the biggest topics under discussion at the NATO summit in Chicago later this month. As officials debate whether to continue to adhere to the Lisbon goals, set at the end of 2010, it is important to understand how we come to understand the war and how we’re not sure if we’re progressing toward our goals.”

The paper poses important questions: “What is success?”, “What are the aims of the war?”.

“10 years after U.S. forces invaded Afghanistan, we still lack the means to tell whether the war is being won or not.”

It criticises the US government for “strategic incoherence at its most stark” for its “worrying ambiguity about the war’s strategic goals and desired end-state” and bemoans the lack of useful indicators, or metrics, that would give clues as the likely direction of the ISAF campaign.  The paper then breaks down President Obama’s three main strategic objectives…

• Deny Al Qaeda safe haven,
• Prevent the Taliban from overthrowing the government
• Build up the Afghan security forces and the government so they can take responsibility for the country’s future

 

…in order to introduce what ASP describes as “nine metrics that will indicate whether those three objectives are being met”.  The paper is an update from the original, which was published in October 2011.

Thoughts

The paper is an important one for analysts and strategists alike – what are we trying to achieve, how do we measure it and how will we know we have reached the goal are fundamental questions.  It follows a key analytical theme that has also been pushed strongly by the Center for Strategic and International Studies (CSIS – see “Measures of “Progress” in Afghanistan in the Spring of 2012”)  The opening critique of the ASP paper regarding the incoherence of the US approach is good.  But that doesn’t stop the ASP paper having significant flaws and omissions.

The title of the paper looks wrong.  “Measuring Success: Are we winning?” looks old-fashioned.  It seems to speak only of narrowly defined military victories by the US govt/US military against the Taliban.  This feels strange, given that the paper rightly criticises ISAF’s obsession with battle statistics and “bodycount” and seeks to address much wider economic and political issues.  It would have been a more relevant and understandable title five years, or even two years, ago, when the fighting was escalating and troops were still coming into Afghanistan.  Now, “Measuring Afghanistan’s progress and prospects” might be a more viable, if less eye-catching, title.

One other thing confused me right from the start.  If the incoherence of the US strategic goals is so “stark” (“…very little agreement among senior officials in the Obama Administration about what that goal really is”), then why is the ASP actually trying to devise metrics to measure something that they judge to be so very flawed?  Might it not be better to write-off the US government’s ideas and devise ASP’s own definition of “success”, together with the means needed to move in that direction and the metrics needed and advocate that as a more intelligent way ahead for US policy?

The nine metrics

However, if I, as the ASP do, suspend judgement on the Obama goals and accept them as a fait accompli, the nine metrics suggested by the ASP are a curious mix and feel very incomplete.  They seem like small slivers of information and it was not clear if they were really intended as the nine stand alone “killer apps” we need to have in order to judge a war won or lost, or whether they are meant to fit into a wider package of analytical information needed.

Over half of the metrics offered (Violent Rhetoric from Religious and Community Leaders, Agricultural Production, ANSF Retention, Childhood Literacy and Self-identified Membership of Al Qaeda) feel like very narrow subsets of the real metrics that are needed – i.e. Political participation, Economic Development, ANSF capabilities, Youth and Education and the State of the Insurgency.  They are certainly not “wrong” as such, but these suggested metrics are only fragments of what is needed.

Some of the metrics seem as if they should be merged – Violent Rhetoric surely being a subset of  Political Participation and Shadow Government and Judicial Institutions likewise perhaps the other side of the coin to Provincial and Local Institutions and Accountability (reunited into an “Extent of Governance” metric, perhaps?)

Work with existing metrics or start over?

Maybe we have to work in large part with the metrics we already have, however flawed.  Although the ASP have identified their ideal set of metrics, I sense that it might be too late – eleven years in to a thirteen year war – to hold out the hope that someone might suddenly decide to collect metrics in the way ASP has suggested, let alone pull all the data together in one agreed and easy to use package.

I feel that the ASP paper could very helpfully have made an attempt at a “collection plan” for the information it says we need.  Even if we accept that these nine metrics are the ones above all else that we need – which I do not – there is no sense from the paper as to whose responsibility it should be to collect this information – I guess ideally it should be a coherent single body (US?  UN?  ISAF? Afghan government?) rather than relying on a myriad of well intentioned but fragmented NGO efforts.  But I suspect even the UN, ISAF or the US are scarcely coherent in terms of pulling together and processing information.  And don’t even get me started on the UK…

To emphasise the point it wants to make, the paper comments on the lack of data availability for the metrics it advocates: “As it stands now, we could not assemble sufficient data…”, “After extensive searching, we could not find a single comprehensive study…”, “The data on shadow institutions is badly incomplete.  No one is measuring it…”, “We do not have sufficient data to measure…”, “We do not have sufficient data to show for certain what the state of al Qaeda in Afghanistan really is…”.  Eleven years in, I am thinking that there might be good reasons why this information still doesn’t appear to be available:

• the data isn’t there
• the data is there but not accessible in the open domain (ISAF, for example, collects a lot more than raw “violence” data…)
• the data is there but has not yet been found
• it is too difficult/dangerous/expensive to collect
• the thousands of analysts inside and outside Afghanistan have not yet judged it necessary to have this information
• the thousands of analysts inside and outside Afghanistan judge it necessary but have other data and metrics to process as a priority
• the bodies that should be collecting it (e.g. the Afghan government) are not capable of doing so

This should probably give us clues as to why this information might not become readily available in the years to come, however much we might desire it, even if someone was tasked to start collecting today.  And this at a time when money, interest and analyst levels will surely be starting to tail off – if this hasn’t started already.  Incidentally, how do we actually collect data on “shadow institutions” which, just to be clear, means Taliban control in villages and districts?  The best way currently to get a handle on the districts is to go there – at least, that will remain the best way until the Taliban start publishing their own open source statistics.  And perhaps this is not the sort of thing that the Afghan government would want to give to much publicity to – perhaps even actively hindering collection efforts.  Similarly, when measuring violent rhetoric from religious and community leaders – how many analysts and linguists might be needed for this?

In fact, in some of the metrics the ASP seeks, the availability (or lack of availability) of the information is probably a metric in itself.  In other words, if we get to a point where the Afghan government is actually capable of measuring the likes of childhood literacy, violence, provincial and local institutions, etc, then we perhaps don’t need to look at the data per se to understand that the Afghan government is progressing in a positive direction.  Perhaps the key metric the ASP need to advocate is “Afghan government ability to collect metric information”!

What to do with the metrics when you’ve got them?

The paper scores some easy wins in the introduction and scene-setting.  US strategy is incoherent.  Check.  We don’t know how to measure progress.  Check.  These are the elements that we think should be measured.  Check.  But after this, its case is less clear.  Collecting this information may entail insurmountable obstacles, as I hope I have just suggested.  Analysing it coherently – to a useful timeframe and to agreed standards – will pose additional and significant hurdles.

The paper acknowledges some of the data that it wants is already out there – childhood literacy, violence, agricultural production, ANSF retention.  It then slightly undermines its case by saying how difficult it is to actually use the data:

Agriculture: “All told, agricultural production is one of the most closely-tracked metrics.  Despite that, the data one might use to draw conclusions is deeply flawed, and does not seem to obey clear trend patterns useful for analysis”.

and this:

Childhood literacy: “The Brookings Institution compiled several studies…UNICEF claims a 49% literacy rate…but these data are sourced to an unknown UNESCO survey…The most recent UNESCO country programming document, from 2010-2011, uses data from the Afghanistan National Vulnerability Risk Assessment Report…It is unclear how these statistics can be reconciled…”

and this:

Violence: “Methodologies for measuring violence vary tremendously, and official agencies disagree about how to observe it.  In September [2011], both ISAF and the UN released conflicting reports of violence in Afghanistan…Different sources compile their data differently, with no set definitions of what constitutes combatants and non-combatants…”

In short, whatever metrics the ASP think is necessary, it is likely that they will have these sorts of problems with the information (if they can get it in the first place).

In a new update of the paper, perhaps the ASP could take a pragmatic look at what information is realistically collectable and how the information could best be used if they are able to get it.  Maybe we have to resign ourselves to making use of the data that we do already have, or can easily collect in the future, and use this more intelligently?  I would also be very interested in ASP attempts to package metrics into some form of an analytical system – a set of traffic lights for each key metric or perhaps a more complex scoring method…?

The nine metrics – a passing comment on each

Political participation – I agree with the need to measure this.  The benchmarks would be very subjective, however – what constitutes a good election turn-out these days?

Violent Rhetoric from Religious and Community Leaders – I think this should be a subset of “Political Participation”.  How would you define and measure this – analysts and linguists are hard to come by…

Shadow Government and Judicial Institutions – I think this is part of the same theme as “Provincial and Local Institutions, and Accountability”.  I agree that government reach (or lack of) should be measured.

Provincial and Local Institutions, and Accountability – see above comment.  I think a wider look at central and local government might be better.

Agricultural Production – ASP paper notes the large amount of data that exists but which is flawed.  Agreed that this is a useful indicator, but it looks to be a small subset of an important wider issue – a look at the economy more generally might be more helpful – investment levels, types of investment, small businesses, transport and communications networks, mines and minerals…

ANSF retention –  Yes, should be measured.  This is a small subset of other key aspects of ANSF capabilities.

Childhood Literacy – How useful an indicator is this?  Isn’t it possible for more literate children still to live in unpleasant, oppressive or war-torn countries?  I am guessing children in Iraq and Syria have significantly higher literacy levels than Afghan children.  Schooling can be undone very quickly by a Taliban-style regime – highly literate children will not necessarily get the chance to flourish.

Self-Identified Members of Al Qaeda – I think I’d rather track self-identified members of the Taliban.  I don’t think open source academic institutes are likely to get this sort of information over the next 10 – 20 years.  Incidentally, is this tracking AQ in Afghanistan exclusively, or also across the border in the “safe haven”…?  Those in Afghanistan are more likely to be low-level fighters, those in Pakistan will be higher value leadership.

Violence – Difficult to disagree with the desirability of measuring violence, but a highly simplistic term for a complex and highly politically charged issue?  Blue on Red, Blue on Green, Green on Blue, Green on Green, Blue on White (civilians), Green on White, White on White…The paper itself notes massive problems in agreeing on definitions and additionally wants to measure the political and social effects of violence.  Sounds tricky.

Perhaps I have missed the point here, but I don’t actually think 9 metrics alone (metric: a set of figures or statistics that measure results) are sufficient to measure “success”.  But the key areas I think we should be analysing are:

1. The information war: media/propaganda/perceptions…
2. Popular mood (Hearts and Minds?)
3. The state of the insurgency – leadership, capabilities, morale, intentions.  In particular political engagement of, and communications from, the Taliban
4. The ANSF – capabilities, morale, retention, political influence/control
5. Economic development
6. Development and nature of popular political engagement – political parties, warlords etc
7. Development, extent and nature of central governance
8. Development, extent and nature of local governance
9. Nature and extent of terrorism
10. Ethnic/tribal dynamics
11. Neighbouring countries – dynamics

And I guess I would probably want to devise 9 or so metrics for each of those areas.

In terms of pure data-gathering, other metrics for measuring the progress of Afghanistan that I would offer are:

1. Investment funding coming in
2. “Laundered” money coming out
3. NGOs coming in
4. NGOs going out
5. Asylum seekers getting out of Afghanistan
6. Refugees voluntarily returning
7. Poppy eradication vs cultivation
8. Taliban fighters handing themselves in
9. ANSF members defecting
10. Afghan government’s ability to gather metric data
11. Successful prosecutions in Afghan law courts

Not all this information is easy to get, but some of it has been collected for years.

Final…

The paper is important and interesting, but, to me a little bit of a disappointment as I expected more from it.  It reads more like a “work in progress” and I look forward to seeing it developed further.  As a final parting shot (and thanks, if you’ve made it all the way down this far!), the one fascinating area that the paper mentioned in passing was this.  In the Political Participation section it says:

“If the Taliban, for example, decides it would rather oppose the Karzai government through running for office and holding protest marches, that is a far better end-state than continued fighting…”

For me, a key exam question du jour for all analysts should be this:

“How can we get the Taliban engaged in politics?”