Unlocking Insights: How Individual Criminal Records Can Transform Research and Policy

If you are suspected of, arrested for, charged with or convicted of a crime or crimes in England and Wales it goes onto the police national database (PND). The PND covers the type of offence, location, date and time(s) when the offence(s) were committed, if that information is known. If you are convicted, or admit guilt, the criminal justice outcome is also recorded, with a few exceptions relating to ‘out of court disposals’. There is a good deal more in the PND, but even those few things I’ve mentioned can be used to great effect by policy-makers but aren’t at the moment because it’s very difficult to access and use PND data outside of police forces or a few teams inside the government.

What can we use this information for?

There are four main topics I’ll cover: understanding offending and types of offenders, predicting future offending, assessing policy effectiveness and multiplier effects. I link to research that used PND either on its own or linked to other data, but don’t mistake this as the norm - these are often one-offs but this sort of work should be much more routine but it won’t be unless we make it so.

Understanding offending and offenders. This is a necessity for effective criminal justice policy-making but one that is neglected. Policies, particularly prison and rehabilitative ones, are after-the-fact attempts at incapacitation or ‘cures’ rather than trying to prevent crime in the first place. However, if we look at entire populations of offenders - such as those serving short-term sentences for hate crime - we can derive new insights into that type of crime, those who might commit it and what to do about it. For example, those imprisoned for violent hate crimes tend to have more prolific criminal careers than non-violent offenders, be older and typically have been to prison already. That means there will be multiple opportunities to try and change their behaviour. This also tells us that there won’t be an obvious ‘profile’ of a person who is imprisoned for hate crime but that people who have wide versatility in offending may have a higher propensity for hate crime, but we can also investigate further to understand if there are any ways of identifying these offenders earlier in life. At the other end of the spectrum, again using population-level data, we can look at young first time entrants to the youth justice system - a key transition into the CJS - to understand how that group changed over time in terms of their offence profile (more serious), look at spatial differences in FTEs (high), whether it’s possible to identify when rates changed (yes), and provide definitive answers as to why they changed (no).

Beyond these applications, centrally held criminal records also allow us to investigate and develop ways of thinking about crime as emergent patterns of behaviour that can begin very early in life. International evidence suggests that there are often distinct offending trajectories relating to the onset, frequency and timing of criminal acts, including some that persist through one’s life. Through linking with educational data we can look at how early education, or risk factors such as school exclusion, relate to different types of offending such as violence. This is useful because it helps us to think about the different policy responses that might be necessary and when they might be best placed to start - not forgetting that it’s ‘never too early, never too late’ to prevent future criminal behaviour.

Predicting future offending. Thanks to decades of research we can now do a lot better* than chance at predicting the future likelihood of crime using only a few pieces of information from criminal records, but this is still far from perfect and only works at the level of a group average over a pre-defined time-frame. So getting a score saying ‘high risk’ might mean, for example, that 70% of people with the same recorded characteristics and situation as you will, on average, commit another crime in the next year. It doesn’t mean you will. But tools producing predictions need to develop, otherwise they become less accurate as the nature of offending and the population changes over time. Again, working at population level with offence data we can incrementally update tools rather than taking decades to make changes that may make practice worse.

What’s coming - is already here - is the use of more advanced statistical techniques and “AI”. The combination of administrative data from different sources and unstructured data from those or other sources, combined with large-language models, mean that we may be able to add a few more percentage points to our predictive accuracy. At the same time, we also have to benchmark the ethics, performance and proportionality of these approaches as they develop; least of all so we’re not sold a lemon by companies claiming to be able to ‘do prediction’ better without the evidence supporting that. It is much more difficult to do any of that if core data sources are largely inaccessible.

Assessing policy effectiveness. Perhaps the most well recognised but still under-exploited research benefit of using PND data is that it allows us to evaluate policies and practices. This has been happening for decades through initiatives such as the justice datalab, and one-off evaluations such as this on burglary prevention, but it’s still far from routine. This means that policies in criminal justice aren’t rigorously evaluated, meaning we may have policies that are ineffective or make things worse (e.g. here). (This problem is exacerbated by the lack of experimental studies in UK criminal justice - something that is changing thanks to the efforts of the Evaluation Task Force - see for example this study on offending behaviour in prisons.) But beyond basic questions about effectiveness on average, using PND and other criminal justice data moves us towards the idea of more precision in policy-making; i.e. what works for whom? Opening up PND to allow for more research and reassessing evaluations already conducted can have several benefits, for one, allowing us to identify whether programmes have unintended consequences.

Multiplier effects. With better access to PND data and by working closely with practitioners and policy-makers on R&D, researchers can help to develop tools and understanding faster, and possibly save the state money. Similarly, better access will allow for innovative solutions to be developed to understand risk - such as work showing the importance of co-offender networks in transmitting risk, but also how to target interventions better. More broadly, the UK is fortunate to have several birth cohort studies, starting in the 1950s. A new birth cohort study is beginning, and with it the benefits it will bring to understanding how generations change. These studies allow us to answer novel policy-relevant questions about how attitudes and behaviour change as children turn into adults, such as knife carrying, or extreme political attitudes. We can do more if we routinely link these studies to administrative data for research. When combined with representative data from surveys or the census, we can also plan for much faster and slower changes that may affect crime rates that are driven by changing attitudes or demographics. For example, there have been sustained moral panics about youth crime in the 1990s, 2000s and 2010s, while at the same time youth crime has dramatically decreased, along with many types of crime more generally (notably homicide). Similarly, demographic changes, such as short-term spikes in male:female sex ratios may influence crime rates, as this study using Swedish registry data explored. Sticking with the Scandis, combining population data in different ways means we can investigate the development of crime from different perspectives. Low resting heart-rate (<60 bpm) during adolescence is predictive of crime, in particular violent crime. It’s not necessarily because of a low RHR that someone goes on to commit a crime or be violent but through knowing this association we can work to understand what else might be important.

What next?

For nearly two decades there has been a gradual reduction in the volume and scope of centrally funded or conducted research on the effectiveness of the criminal justice system in the UK. This is because analytical functions inside government were cut, as was funding for research and evaluation, even on core Areas of Interest (which don’t come with funding). To get the best from the hoards of data the justice system generates as it functions, we need to vastly simplify access to and working with PND and other criminal justice data, ideally allowing researchers access without the need to burden government teams to supply data, and to provide training for future generations of analysts in and out of government how to work with these data sources for the benefit of public policy.

*****

*about 70% predictive accuracy versus 50% chance of being right through flipping a coin.