Tony Thorne was one of the officers of the Apollo project and advised the team on the task of merging large amounts of data. Thorne, a former counter-terrorism officer with the Wales Extremism and Counter Terrorism Unit, said he was shocked by what he saw in Scotland. “We left Scotland with a process that was in no way complete or adequate,” he said.
The main issues that emerged during the test run resurfaced right away, according to emails and internal memos from 2014 and 2015 reviewed by BuzzFeed News.
Officers described a system that “routinely crashed” and “timed out after 10 minutes,” with failures so severe that they dramatically increased “the time it took to complete a simple task.”
Even simple searches caused problems. An agent had described how he entered a search term and got a result that was too broad. He started manually sifting through the documents to find out which ones he really needed – but while doing so, the system crashed. When he logged back in, he typed in the same search terms and found that “the search result was not the same”.
Officers using the new systems also reported serious problems with the problem the NCIA wanted to solve: communicating with other troops and agencies. After a suspect entered the UK by plane, an officer reported that they had received an important intelligence report from officers at the airport in an illegible format. Another told a member of the Apollo team that the NCIA’s inability to share intelligence with the other regions still using the old system was a critical risk that “could lead to intelligence failure.”
The quality of the information that did enter the system was often poor. In some cases, the NCIA was inundated with irrelevant information; in others, vital intelligence never showed up to the NCIA. An officer complained that the system “automatically records” documents that had nothing to do with terrorism. “This issue was something that was always talked about,” the officer wrote, “but now that we’re live there, nothing seems to have been done about it.”
The NCIA is built on the template of a pre-existing system called the Home Office Large Major Inquiry System, four sources told BuzzFeed News. The problem, one said, is that HOLMES is used to investigate incidents that have already taken place, while the NCIA is intended to prevent attacks. Another officer told BuzzFeed News that building the NCIA on top of the HOLMES system caused defects that made large amounts of intelligence difficult to find.
Officers echoed these concerns in their emails and official reports. One of the key features borrowed from the HOLMES system was a search tool, much like Google, that should enable officers to quickly retrieve documents containing a particular word, regardless of where on the record the specific word appeared. If it worked, it would make it much easier to find specific information about potential terrorists from hundreds of thousands of files.
But the search function didn’t work. Officers found that if they entered the same search term multiple times, they would often get a different result each time. The search function also couldn’t scan for dates of birth, making it much more difficult to find the right document.
This shortcoming was linked to another major problem. It became clear early on that many duplicated records would make their way to the NCIA — as it collected data from multiple troops who often held the same file on a particular individual. An internal report seen by BuzzFeed News acknowledges that doing so would create a “knock-on” effect that hampered analysts. But senior people eventually decided that “there would be no deduplication” until the entire UK used the NCIA.
A Manchester-based officer who later began using the NCIA told BuzzFeed News that finding what you were looking for is “trying to find a needle in a haystack” — such a struggle that “you could miss vital intelligence clues.”
Thorne, the counter-terrorist who worked on the NCIA, became increasingly concerned. “Unfortunately,” he wrote to colleagues in a February 2014 email, “as we are all fully aware that the NCIA has struggled to deliver what was promised and was not fit for purpose.”
The rollout of the NCIA continued.