British Police Built a Sprawling Crime-Prediction Machine. Some Results Couldn’t Be Trusted

British Police Built a Sprawling Crime-Prediction Machine. Some Results Couldn’t Be Trusted

As UK police embrace the AI revolution, a WIRED investigation reveals the messy inside story of one region’s experiment with predictive analytics.

The Think Family Database holds records on close to half a million people who live in the city of Bristol, England. For many years, few of them knew anything about it. Launched in 2016 by the Bristol City Council and the regional Avon and Somerset Police, the database has stored all manner of sensitive information-police intelligence reports, housing status, mental health records, teenage pregnancies, enrollment in parenting courses, free school meals.

On top of this sensitive data, officials built machine-learning models to assign scores to thousands of adults and children. They hoped to build what they called a “picture of threat, harm, and risk” in the region. At an event in early 2022 to help officials tackle child exploitation crimes, one police data scientist described part of the approach this way: “I essentially dump all that data in a big bucket and stir it with a data-science spatula, and we come out with a lovely risk score for everybody.”

This risk scoring inside the Think Family Database was just one part of Avon and Somerset Police’s sprawling predictive analytics program. Among at least 23 separate models the force created were algorithms to identify the risk that people would commit burglary, fail to turn up in court, go missing, or become a victim of domestic abuse. One senior officer described creating a “league table” of the area’s most dangerous criminals-an apparent reference to the Offender Management App, which was designed to hold data on around 300,000 people in the region.

How the police have developed and used their predictive tools hasn’t always been clear to the public. John Pegram, the leader of a local police accountability group in Bristol, says he didn’t hear about the Offender Management App until 2023, years after it had been created. When he did learn about it, he began to suspect he might be included. “I think I knew I was on the app,” Pegram says.

In early 2024, Pegram filed a request to find out how the police were using his data. The police refused to say. Months later, after Pegram had hired solicitors to work on his case, the police confirmed he was on the app but declined to elaborate further. Like others across Bristol, the UK, and, increasingly, around the world, Pegram didn’t know whether he had been scored by an algorithm, what that score might be, or how it could affect his interactions with the authorities.

The Investigation

WIRED, working in partnership with the nonprofit newsroom Liberty Investigates, plus the Bristol Cable and Lighthouse Reports, obtained hundreds of pages of documentation from public records requests to build the most comprehensive picture to date of Avon and Somerset’s regional experiment with data collection and predictive analytics. (Liberty, the parent organization of Liberty Investigates, had some early involvement in a potential legal challenge to the program and continues to support Pegram’s litigation.)

The investigation reveals that at least two of these risk-scoring models were quietly abandoned after Bristol City Council staff deemed they could no longer trust them. Previously unreported documents show government inspectors and independent reviewers highlighting a startling lack of transparency about some elements of the program and warning that the systems could undermine public trust. Police data disclosed to WIRED-comprising more than 36,000 model performance scores-appear in some cases to show “genuinely poor predictive performance,” according to an independent analyst who reviewed the data for WIRED.

These findings come as the UK appears poised to embrace predictive analytics and artificial intelligence across the criminal justice system. A familiar face is helping lead the charge: the former chief constable of Avon and Somerset, Andy Marsh, who now heads the national standard-setting body for forces across England and Wales. As CEO of the College of Policing, Marsh has said that effective AI should be “injected like heroin” to speed up British police work. In a recent interview, Marsh said his organization was examining around 100 currently deployed AI tools, including for predictive policing. “Our job is to test the ones that work properly, test them with rigorous evaluation, and then spread them like wildfire through policing.”

The Origins

In 2014, Avon and Somerset Police was under pressure on multiple fronts. The force, like others across the UK, had seen its budgets slashed. Its chief constable had been suspended. An official report had highlighted its failure to stick to procedures to protect some victims of domestic abuse. After that report was published, the force’s head of performance said, “We believe predictive analytics is the solution.”

Gary Davies, a former police chief superintendent who had moved to a role at the Bristol City Council two years earlier, was thinking along similar lines. Davies led a team at the council supporting children and families. When families were in crisis, “it was blatantly obvious,” he says. It was much harder to spot those who were at the top of a downward spiral. Davies believed the answer lay in data.

A child’s school might hold a record of increasing absences, while the police might know if the child had recently witnessed domestic abuse for the first time. On their own, these might not be enough to trigger an intervention from social services. But together? “If you could see the whole picture, you would realize that the trajectory they were on was going in the wrong direction,” he says.

Starting in 2015, a small group of Bristol City Council and Avon and Somerset Police staff moved into one of the city’s police stations to work on a solution to that problem together. The Insight Bristol team, headed by Davies, started pulling together data from across the public sector to provide frontline workers with all the information they might need about children and families.

Data and Consent

The Insight Bristol team didn’t seek residents’ consent to use their data in the Think Family Database. Instead, Davies explains, the team relied on “legal gateways”-a term that describes when data sharing is deemed necessary to meet an agency’s legal obligations, such as the need to protect children. “If you were to give the impression that people had consent, then it creates a false illusion, because, actually, as [a] local authority or police or whoever, we have to keep those records.”

Initially, residents could not opt out of the database; later, the council included an opt-out option in its tax letters to residents. Davies, who recently retired, believes the project did help protect children. “It improved the understanding of risk and vulnerability for children and families,” he says. “It provided that information in a far more efficient way.”

When it came to communicating that to the public, Davies says, “it was fairly difficult to get any enthusiasm or interest from groups of people.” Those who did engage said they understood the need to use personal data, he recalls, summarizing the feedback as “We don't mind you using it to support us, but we don't want you to use it against us.”

Ethics and Early Warnings

While the Insight Bristol team was busy creating the Think Family Database, Avon and Somerset Police had begun exploring the potential of predictive analytics. In March 2016, the force’s ethics committee met to consider how the work should proceed. Members advised that “careful consideration had to be given to what data is used” and “the variables that are used in the process,” concluding: “The use of the system must be treated with some caution and it must be ensured that there is no bias.” The committee advised that, if the force’s predictive analytics work was to proceed, “the public must be informed as to why and how you are carrying out such processes.”

Once work to compile the Think Family Database was completed, a police data scientist spearheaded the development of predictive risk models for the project. One of those models aimed to identify children at risk of sexual exploitation. The CSE model, as it was known, drew on a wide range of datasets held by the police, council, and other public agencies. Barnardos, a child protection charity, provided anonymized data involving 1,000 children known to have been sexually abused. The model was designed to detect children with similar characteristics.

The system analyzed children’s social connections to determine if they were linked to anyone else deemed vulnerable to, or a likely perpetrator of, exploitation. Being flagged as “in need,” “persistently” absent from school, or having mental health concerns would increase scores created by the CSE model, documents say.

The police’s efforts to draw on such a wide range of data raised early concerns. In 2018, researchers at Cardiff University’s Data Justice Lab reviewed several UK citizen scoring programs, including the work in Bristol, noting, “The variables being used can in practice be proxies for poverty.”

Davies recalls that most of the children with the highest risk scores were already on the authority’s radar. “They were complicated children already being worked with by social workers and family workers,” he says. “Most of the output told you what you already knew.”

Still, the police’s enthusiasm for predictive analytics was undimmed. “We want to make choices today that will prevent the crime from happening in the first place,” a police business intelligence manager said in 2018.

A model for predicting child criminal exploitation, which according to Bristol City Council was introduced in 2019, again drew on data from a wide range of public agencies, including whether a family was receiving housing support or in rent arrears and if a child was receiving free school meals. That year, chief constable Andy Marsh announced: “In 12 months every part of Avon and Somerset Constabulary will be driven through predictive analytics and visualization.”

Mixed Results Elsewhere

Across the UK, other law enforcement agencies had been conducting their own experiments in predictive policing-with mixed results. Kent Police, the first UK force to test the technology, had recently canceled its contract with the US firm PredPol, saying it had been “challenging” to show the program reduced crime. Durham Constabulary had come under fire for using sociodemographic data to try to predict risk of reoffending.

By comparison, Avon and Somerset Police looked like a success story. Behind the scenes, though, the police’s predictive analytics work seemed “messy,” according to Elle Pearson, a researcher at Royal Holloway University of London who is completing a PhD studying the programs. “When I started, nobody could tell me what data they had or where it came from or what system was using which data,” says Pearson, who has interviewed more than a dozen staff members from the police and local councils.

Over time, Pearson observed a “function creep,” with systems becoming more expansive, combining more data, and spreading beyond their original purposes. While “conscientious” teams oversaw the development of data analytics, Pearson says, there was little transparency. “In some instances it might just be one person who’s creating these risk models that are making decisions affecting potentially hundreds of thousands of people,” Pearson says.

Reviews and Abandonment

By 2021, according to a review of Insight Bristol in government records obtained by WIRED, officials from the Centre for Data Ethics and Innovation (since dissolved) were hearing of “ethical tensions” associated with the project. “Large amounts of sensitive data” had gone into the risk scores, the reviewers said, noting that it had been gathered using “legal gateways” rather than by building trust with local people. “Legality is not the same as legitimacy,” they said.

Two years later, the nonprofit organization Social Finance conducted an independent review of the Think Family Database and Insight Bristol’s data work. The 100-plus-page review, which appears to have been made public only after a records request from WIRED, was commissioned by Bristol City Council and the nearby Somerset Council, which was planning a similar system.

The review found that the Think Family Database and its visualizations had been useful for child protection staff and could “lead to timelier responses.” However, it described the risk-scoring models as the “weakest element” of the project, noting that a “lack of accuracy” had undermined their potential to be useful. Council staff had raised doubts about the models designed to assess risk of child sexual exploitation (CSE) and child criminal exploitation (CCE).

The review was completed around the same time the council stopped using the models, which staff had recently described as “not fit for operational use.” While previously the CSE and CCE models had largely been telling staff what they already knew, according to Gary Davies, the former head of the Insight Bristol team, those same social workers told the Social Finance reviewers they increasingly found the algorithms to be “inaccurate.”

In an email about the CSE model, one staffer had written: “There are people who’ve been victims of sexual offenses in the last month scoring below those who have been perpetrators of burglaries.”

According to the Social Finance review, there was a reason for the sudden drop in perceived quality: Police had stopped using Bristol City Council data. Officials wanted to “profile children across the entire Avon and Somerset Police boundary”-five separate councils-with the same algorithmic approach, the review says. While the force tried to strike data-sharing agreements with other local authorities, those efforts stalled, meaning the only data it could use to develop the algorithm was its own. This would include perpetrators and victims of crime but not the array of sensitive social factors the model had drawn on before. After the switch, Bristol City