The Nigeria Security Tracker: How (not) to Design an ICT-Enabled Project

For those interested in Technology for Peacekeeping consider taking our online certificate course, Technology for Conflict Management and Prevention, starting July 23rd.

**Disclaimer: These are Asch’s personal views and do not represent those of his employer.

You’ve heard of the 90/10 rule, right? I hadn’t heard the concept, at least, until recently. The meaning, though, I learned the hard way—an ICT-enabled project should be 90 percent planning and only 10 percent digital tool.  Not the other way around.

We initiated the Nigeria Security Tracker, an effort to catalog and map political violence based on a weekly survey of domestic and international press, at least two years ago. We wanted to answer the question “are things getting worse in Nigeria?”

The death of Nigeria’s president in office with upcoming elections, an increasingly divided electorate, and an apparent up tick in violence in the north and the middle of the country raised serious doubts about Nigeria’s stability.

And yet many disagreed. The optimists said things were getting better; the pessimists that Nigeria was becoming a “failed state”; and everyone else that Nigeria would continue to “muddle through,” as it had done since independence in the 1960’s.

Measuring levels of violence seemed like it could give us a more precise answer. While the Nigerian press has many shortcomings—lack of journalist training and professionalization, concentration of ownership and coverage in the south—it is relatively free, and there is a lot of it. Also, the presence of major outlets like Reuters, BBC, AP, AFP, and the Wall Street Journal added another layer of reliability. Indeed, some of the best Nigeria analysis I’ve seen comes from open sources. You just need to learn to read between the lines.

Mapping seemed like a useful and visually engaging way to organize our information. But without funding or any programming experience, our options were limited. We experimented with manually pinning incidents to Google maps and embedding on our blog. We tried to pitch Ushahidi to the web department hoping to get programming support, but without success.

Eventually, we abandoned the project–until Crowdmap was launched. Free, hosted on Ushahidi servers, preprogrammed, and simple to set up and use, it made the security tracker possible.

Our Mistake

We designed our research methodology around Crowdmap capabilities. We could include basic descriptions of events, and simple codings as well as details like causalities, but it was labor intensive.

Not until three months later, when we sat down to review our work, did we realize the shortcomings. Putting incidents on a map is useful if you want to see where violence is happening. But less so when you want to know when incidents occurred, or if you wanted to look at trends or correlations over time. (The automation feature of the map, while fun to watch, is not a terribly useful analytical tool.) We discovered it was impossible to look at, for example, the relationship of violence perpetuated by the security services and causalities.

Fundamentally, the project was supposed to be about the information captured. Not the technology. And we had it backwards.

We also discovered a major lost opportunity. Because we tailored our methodology to what could be included on the Crowdmap, we failed to capture other useful information, such as attacks on religious establishments, which only required marginal extra effort.

The Redesign

Given that our project was supposed to answer a particular question that a map alone couldn’t, we relegated the Crowdmap to a component of the project–no longer the driver. We thought more thoroughly about what kind of information we needed, and what we could glean from press reports of political violence.

Fortunately, we had documented all of our sources. So we could return to our original information and recode, albeit with a significant time commitment.

The Way Forward

Despite better planning the second time around, we continue to find shortcomings. We defined one variable, “sectarian violence,” inadequately, which means coding has been inconsistent throughout the project, making it less useful.

We are also vulnerable to technical problems on Crowdmap’s end. When there is a bug in the system, there is nothing we can do but wait for Ushahidi to fix it. (Recently, the site was down for about a week.)

Finally, we now have a year’s worth of information. It’s a huge dataset. And we still haven’t figured out what to do with it. Yes, we can make defensible, conservative estimates of causalities caused by actors like the police or Boko Haram. We can also show any escalation or decline in violence across the country.

But this only scrapes the surface of what our data can tell us. Admittedly, this is a good problem to have.  But given the time and resources we have already committed, and the wealth of date we have accumulated, we are constantly trying to balance benefits of the security tracker with the costs of maintaining it.

Asch Harwood is a specialist on Africa at a New York City-based think tank. 


avatar



  • Jeff Takle

    One of the most common mistakes with data visualization, I think, is to apply the wrong tool for the job. Generally speaking, maps are great for consuming finished information pieces…the punch line in an analysis but not for data “exploration” (hardcore ESRI GIS folks, excepted) which it sounds like was the initial goal for the project.

    One way our team has been addressing this issue is to think about — and provide — our clients with three distinct data visualization tools. First, is the project team’s status board that mostly monitors the activity, incoming reports, and more M&E type stuff. We find that a hacked up Excel (pulling real time data connections from external databases) or Google Spreadsheet / Google charts often is sufficient. These dashboards usually aren’t publicly available and they ride on free/existing software.

    The second tool is for data exploration. For this we recommend tools like Spotfire because they’re great at showing correlations and allowing users to quickly mash up all kinds of data combinations rapidly. The goal is to discover correlations and causation. But, because it’s expensive and a more complicated tool to use, we typically don’t make these visuals available to the public; they’re used by a data scientist or statistician along with project SMEs to explore and look for the gold. Small team, specialized purpose.

    The third toolset is for public consumption and sharing the punch line. For this, tools like Crowdmap, Ushahidi, Socrata, Tableau, and sometimes even Google Charts. While data wonks like sliders, filters, and sorting, most general consumers — we find — just want to see the punchline. Simple, clean, and elegant. Shareable widely and data is downloadable if somebody wants it.

    Not that the model is perfect, nor are the tools, but what do you think? Can this segmented approach (project staff, data pioneers/explorers, and public consumption) help focus the technology decisions better? Would it have changed your approach on this project?

  • Collin Sullivan

    Hi Asch,

    Thanks for sharing these lessons learned, and +1 to Jeff’s comment above that it’s important to choose the proper medium to represent the data’s punchline, rather than shaping the data to fit a medium (map, graph, etc.). It seems you learned that lesson, too, and I appreciate your explanation of the mistake and steps you took to correct it.

    I think your experience is also a great example of why it’s a good idea to always consider the desired outputs of a data collection and analysis project early on in the design phase. Working backwards from those goals is a great way to identify which questions need to be asked during interviews or data collection, and which information needs to be gathered, from whom, and how.

    I wonder if you could speak to the question of selection bias, as it’s something that was not mentioned in your post. I think maps are really fantastic for visualizing the geographical elements of a dataset, and that can be very useful with human rights or conflict data. It’s important to specify, though, what the data is not. In your post, you write that, “Putting incidents on a map is useful if you want to see where violence is happening.” Perhaps a more precise statement would be, “Putting incidents on a map is useful if you want to see where reports are indicating violence is happening.”

    This is an important distinction, and it’s less about being nitpicky about wording and more about understanding what the data are saying so we can better use the information. Looking at the screenshot of the Nigeria Security Tracker included in your post, at first glance it appears to indicate that more violence occurred in the north central and northeast regions of the country. That might be right, or it might be skewed because the data collection methods favored reports coming from those regions. Or because victims trusted the reporters more in the north central and northeast. Or because the people in other regions are too afraid to report. Where the data are coming from, how the information is being reported and how those reports are collected can all serve to skew a dataset away from reality.

    In our experience organizing data for quantitative purposes (over 20 years in over 30 countries), we find that direct reporting is always statistically biased.

    If we start from the premise that all datasets are incomplete–or, to put it differently, that all datasets are samples–then the next question must be about whether our sample (our dataset) is representative. The only way a sample can be known to be representative is if it has been chosen randomly from the affected population, and that’s essentially impossible in human rights work.

    There is still much to glean from a non-representative sample; it can tell us a lot. But it cannot tell us the statistical patterns and magnitude of events the way a representative sample can, so we have to know which it is before we can know what the data are actually indicating.

    I don’t know your methodology, or if or how you corrected for bias. Do you use a model that estimates the probability of each report? You may well have, and if so it would be really great to learn more about how you did that. If not, it would also be great to hear about whether and how you might redesign your approach to the security tracker.

    Thanks again for your post, Asch. Looking forward to hearing your comments.

    Collin Sullivan

    Human Rights Program Associate

    Benetech

    • Asch Harwood

      Dear Collin,

      You are absolutely correct. Selection bias is a major issue, particularly in Nigeria. Indeed, whenever we discuss our findings, we are very careful to go over the shortcomings of this methodology. We emphasize that our data is indicative, not definitive.

      And actually it goes both ways. The press is mostly based in the South. But the extent of violence in the North has attracted a lot of interest, which means there is the impetus to cover it. However, both security services and Boko Haram have put pressure on journalists, which makes covering the violence dangerous.

      In general, this project is only part of our research into Nigeria, and generally our experience suggests that what you see on the map is conservative. Casualties numbers are often much higher, incidents in rural areas go unreported, not to mention possibly manipulation of press reporting.

      We actually suspect that it is the Niger Delta that is more under reported than other parts of the country.

      Asch

    • Asch Harwood

      Dear Collin,

      You are absolutely correct. Selection bias is a major issue, particularly in Nigeria. Indeed, whenever we discuss our findings, we are very careful to go over the shortcomings of this methodology. We emphasize that our data is indicative, not definitive.

      And actually it goes both ways. The press is mostly based in the South. But the extent of violence in the North has attracted a lot of interest, which means there is the impetus to cover it. However, both security services and Boko Haram have put pressure on journalists, which makes covering the violence dangerous.

      In general, this project is only part of our research into Nigeria, and generally our experience suggests that what you see on the map is conservative. Casualties numbers are often much higher, incidents in rural areas go unreported, not to mention possibly manipulation of press reporting.

      We actually suspect that it is the Niger Delta that is more under reported than other parts of the country.

      Asch

  • julia john

    Fantastic learning
    guys! I’m a die heart fan of your respective blog site.
    adt security