Navigating data governance risks: Facial recognition in law enforcement under EU legislation

Abstract

Facial recognition technologies (FRTs) are used by law enforcement agencies (LEAs) for various purposes, including public security, as part of their legally mandated duty to serve the public interest. While these technologies can aid LEAs in fulfilling their public security responsibilities, they pose significant risks to data protection rights. This article identifies four specific risks associated with the use of FRT by LEAs for public security within the frameworks of the General Data Protection Regulation and Artificial Intelligence Act. These risks particularly concern compliance with fundamental data protection principles, namely data minimisation, purpose limitation, data and system accuracy, and administrative challenges. These challenges arise due to legal, technical, and practical factors in developing algorithms for law enforcement. Addressing these risks and exploring practical mitigations, such as broadening the scope of data protection impact assessments, may enhance transparency and ensure that FRT is used for public security in a manner that serves the public interest.

1. Introduction

Technology has revolutionised law enforcement, evolving from automated fingerprint systems to sophisticated crime prediction tools (Hoey, 1998). Law enforcement agencies (LEAs) now leverage big data and facial recognition technologies (FRTs) to process vast amounts of biometric data for law enforcement purposes, including facial features and retinal patterns, collected via tools like CCTV cameras, drones, and online platforms. FRT primarily handles biometric data that is diverse, such as facial features and retinal patterns, linking them to the purpose of FRT which is to use them for identifying individuals (European Data Protection Board, 2023). In the US, 30 million cameras record 4 billion hours of footage weekly, potentially used for monitoring activities like riots (Hodge, 2020; USGAO, 2021).

FRTs usually have the potential to enhance public safety, security, and justice, aligning with the United Nations Sustainable Development Goals (Chui et al., 2018; Shi et al., 2020). For example, FRTs are employed at the Schengen Area’s external borders to enhance the security and integrity of immigration procedures. Because it is used by border authorities for a very specific purpose (to monitor and control the crossing of a land border), at a specific time (when an individual is attempting to cross the border), and in a specific place (a border crossing point), it may serve to establish a level of trust between the public (the trustor) and the authorities (the trustee), despite the lack of empirical evidence. Nevertheless, such a specific use increases the likelihood that the negative impact on fundamental rights will be minimal and that identifiable risks can be partially mitigated. In this way, it is an opportunity for LEAs to ensure safe migration, which is one of the elements of public security that falls under the broader public interest.

The debate over whether FRT genuinely enhances public security and serves the public interest persists. This is primarily because FRTs also pose significant surveillance risks, with studies showing public perceptions about increased surveillance despite recognising FRT’s crime prevention potential (Pew Research Center, 2022; Kostka et al., 2023; Lyon, 2002). One reason surveillance may proliferate is due to violations of data protection and privacy rights, stemming from the continuous and varied use of FRT (De Cremer, 2020; Floridi et al., 2020). Data collected through FRT can be used to create comprehensive profiles of individuals, including indicators of potential criminality or even categorisations of groups with similar identities (Delikat, 2021; EDPB, 2023). Automatic profiling often not only limits individuals from expressing themselves or correcting potentially inaccurate categorisations but also raises concerns regarding the unintended purposes of the technology, which can lead to public distrust. Such limitations complicate individuals’ efforts to understand, contest, and rectify erroneous or biased decisions (Palmiotto, 2024), therefore decreasing the reliability of the system’s purpose of operation to serve the public interest.

In the EU, the GDPR and the AIA primarily govern biometric data processing through FRTs. The GDPR ensures lawful data processing, while the AIA aims to create trustworthy AI based on such data. Biometric data is classified as special personal data under the Article 9 of the GDPR, with processing allowed only in exceptional cases, such as for public security. Article 5 of the AIA classifies biometric data processing tools under the category of high-risk AI systems meaning that the placement of such tools in the market is possible under strict rules. Furthermore, real-time biometric data processing tools in publicly accessible spaces for law enforcement purposes are classified under the category of unacceptable risk which means that their operation is generally prohibited, except where they serve limited purposes. Obviously, both legislations permit biometric data processing and tools for law enforcement purposes with considerable discretion in interpreting rules regarding public security. Even though use of FRT by LEAs under the AIA will be possible under some heightened safeguards and within the guidance of the interpretation of national authorities of the member states, at LEAs in the interim period between the introduction of the AIA and the first interpretations from the national authorities arrive, LEAs will need to interpret rules on processing biometric data for law enforcement purposes independently. Even after that, the legal, technical, and practical aspects of deploying FRT could lead to breach of data protection rights, affect uniform application of the GDPR and the AIA, and cause inconsistencies during implementation. For this reason, human rights organisations criticised the removal of the proposed general ban on real-time FRTs as outlined in the draft AIA text of June 2023, as well as the final text, which they found to include insufficient restrictions on FRT use. They argued that these provisions undermine privacy and exacerbate public concerns regarding the balance between security and privacy (EDRi, 2021; EDRi, 2023; EDPS, 2021; Amnesty International, 2023).

This article examines four potential risks posed by FRT in law enforcement, focusing on legal, technical, and practical aspects under the GDPR and AIA. If not carefully evaluated, these risks can erode public trust, undermining the public interest LEAs aims to protect (Mazzucato, 2022) by providing insufficient scrutiny power to individuals to defend their data protection rights. Notably, aside from the GDPR and the AIA, which partially regulate FRT implementation, there is currently no specific legislation in Europe governing FRTs, particularly their application in law enforcement (Mobilio, 2023). The risk analysis presented here is to highlight the gaps and aims to propose a tech-by-tech approach to extending the scope of implementation of existing legislation to FRTs in law enforcement. The expansion of data protection impact assessments to FRTs’ applications, their adoption as part of the decision-making process, and their integration into a wider risk management scheme introduced by the AIA could contribute to lawful use of FRT for public security purposes.

2. FRTs’ diverse applications

In order to engage in discourse surrounding FRTs, it is first necessary to define the term. This is a challenging task given the lack of a universally accepted definition. The extant research and legal discourse present a multitude of technical and legal interpretations of FRT, each emphasising different aspects. For example, from a legal perspective, as seen in the AIA, FRT is conceptualised primarily through its role in biometric data processing, as will be presented in the definitional challenges title of this article later. On the technical side, for instance, Selwyn et al. (2024) emphasise the various functions of FRT, such as facial processing, categorization, and analysis, with each technique being specific to its intended use. Despite the absence of a universally accepted definition of FRT, all definitions converge on a common focus: the purpose of use. This approach aims to reduce the ambiguity surrounding the term and address the issue of the lack of a standardised definition. Nevertheless, while the purposes of using FRT can sometimes be unclear, the associated risks are not, as Akbari (2024) stated. This means that from a risk assessment perspective, it is the intended use, not only the definition, that determines the risk. The purpose of use shapes the definition, and as the purpose changes, the risks change accordingly.

FRT serves various purposes, such as face recognition, emotion and behaviour assessment, and individual tracking in crowds (Gillis et al., 2021). These purposes can be classified into three main categories with each point a function: authentication, identification, and profiling. In law enforcement, one or multiple functions could be operated, and the discernment of functionality and purpose remains pivotal for comprehensive risk assessment (Christakis et al., 2022b). The methods used to develop FRT algorithms typically reveal their intended function and purpose. For instance, a one-to-one search method is employed for authentication purposes, such as unlocking a mobile phone with a facial identification tool. One-to-many search in airports is used to identify individuals and allow them to cross the border. These methods rely on a query to a database of known individuals and work post-biometric processing. The many-to-many search method used in live biometric processing activities is particularly noteworthy due to the heightened risks it introduces. This has the potential to be used for surveillance purposes on anyone, anywhere, and at any time. Surveillance purposes here could include not only recognising someone, but classifying them according to their age, gender, or race, or even further categorising them according to how they feel. According to a study, out of the 27 EU member states, 11 currently employ FRT, with the one-to-many method being commonly used by the LEAs (TELEFI project, 2021). In Europe, the Metropolitan Police Service in London has a track record of employing live FRT that was developed using a many-to-many method (Mansfield, 2023). Nevertheless, despite its ability to process a facial image with a time delay of a few seconds, a live FRT is currently being used by police in the German eastern state of Saxony and in Berlin in the area of cross-border gang crime (Borak, 2024).

According to the ICO (2021), risks arise when the same functionality is used for different purposes and in different settings, or when the same purpose could be served by different functionalities. Christakis et al. (2022b) note that when combined with techniques such as predictive policing, FRT becomes even riskier as it combines the potential risks of two different systems. This could go beyond the mere public security purpose to lifelong surveillance. From the legal point of view, the GDPR does not differentiate between the methods mentioned technically (Kindt, 2020, pp. 62-69), which creates uncertainties particularly for providers who focus mostly on the technical aspects of FRT in their efforts to ensure public security. Further, as the methods also give some ideas on the purposes, the risks arising from specific uses of FRT differ. Similarly, LEAs may struggle to comply with the law because they may not always be aware of the technical perspective of the providers. Failure to adhere to legal and ethical standards by the key stakeholders involved in FRT operations is likely to undermine public trust and acceptance, as will be shortly discussed below. Technical complexities, coupled with legal ambiguities, pose significant challenges to upholding fundamental principles outlined in the legislation, such as purpose limitation, data minimisation, data accuracy, and administrative constraints. Each of these factors plays a crucial role in transparently communicating the intended purpose of FRT deployment to the public. The GDPR, like the AIA, avoids regulating specific technologies and instead establishes a general framework. However, as AI advances and becomes more specialised, the associated risks will increasingly target specific sectors. Therefore, a technology-specific approach to risk assessment should have been considered by the EU lawmaker, even though this is now challenging to implement. Nonetheless, it is crucial to emphasise the need for targeted risk assessments for FRTs, as existing legal tools offer guidance on addressing specific technologies.

3. Intersection of trust, risk, and FRT for public security

In order to ascertain whether a risk-based approach can be applied to a specific technology and its correlation to public trust, it is first necessary to see the relationship between the relevant terms. The concept of public trust represents a fundamental pillar of public interest, as it serves to reinforce the conviction that institutions are committed to safeguarding the collective interests of the community, such as public security which is the main focus of this article. The effective management of these areas is entrusted to state authorities with the objective of safeguarding the social good. Simply, a lack of public trust in authorities represents a significant challenge to the foundations of democratic systems, human rights, and the rule of law. The legal framework defines the scope of public interest, though is not an ambiguous term, and the extent to which the state is permitted to protect these interests. Through the consent and trust of the governed, the state is authorised to protect these interests through legal tools. This section will present the inconclusive outcomes observed in the literature with regard to the positive or negative correlation between public trust and a specific technology targeting to ensure public security. This understanding then provides the basis for evaluating the use of FRT under the GDPR and the AIA from the risk assessment perspective.

The term trust is inherently ambiguous, defined through various dimensions at both individual and public levels. At the individual level, trust stems from psychological, cognitive, rational, or emotional factors, influencing public trust. Trust is contingent on aligned interests, where mutual expectations of trusted and trustee play a crucial role (Hardin, 2001). Palmisano and Sacchi (2024) argue that trust depends on perceptions of uncertainty or risk, with the example of education reducing risk-taking tendencies. Degli-Esposti and Arroyo (2021) highlight that control mechanisms like transparency are essential in fostering public trust in technology by mitigating risk. Hence, trust and risk are interrelated, affecting both interpersonal and technological contexts.

In the context of FRT used for public security, assessing the trust between the public and deployers (particularly LEAs and providers) is crucial to determine if the technology meets its intended security goals. Various studies explore public trust in LEAs, some linking it to governance processes (UNDP, 2021), and many attempts to measure it via public surveys. These surveys, however, reveal mixed results: some studies suggest that trust enhances positive attitudes towards surveillance, while others argue that surveillance undermines trust (Herreros, 2023). For instance, the Police Barometer study shows that 90% of Finns trust their police due to effective emergency operations and fair decision-making (Vuorensyrjä et al., 2023). However, some scholars criticised the complexity of the parameters used in these surveys and highlight a lack of theoretical rigour (Kääriäinen, 2008; Björklund, 2021). Thus, existing literature provides only descriptive insights into the trust-LEA-technology relationship, often failing to confirm its validity. Further, researchers suggest that factors like ethnic diversity, religious groups, education levels (Palmisano and Sacchi, 2024), police treatment of minorities (Valentin, 2019), and trust in justice and institutions (Hough et al., 2013) influence public trust. Procedurally fair treatment, including police interactions and language use, also affects trust (Murphy, 2013). Presumably, it is impossible to consider all these factors simultaneously when evaluating potential risks, however, identifying who benefits most from the FRT should highlight potential imbalances and inform more precise risk assessments.

3.1 Who benefits most and at what risk?

FRT tools for security use present both advantages and disadvantages that can impact public trust in these technologies and their deployers. Clearly defining the purpose, target, and duration of their use can help mitigate public distrust towards LEAs. When used specifically for identification or verification rather than widespread monitoring, FRT has the potential to enhance security in targeted scenarios, thus aligning with public interest by contributing to improved public security. For instance, individuals may perceive an enhanced sense of security due to the monitoring of perpetrators of criminal acts such as bank robberies or acts of terrorism. FRTs can assist in identifying individuals whose documentation is lost in disasters and in identifying victims of war and disasters (Selwyn, et al., 2024). In healthcare, FRTs aid medical professionals in rapidly identifying novel diseases, such as during COVID-19 they helped understand the virus’s spread by providing statistical data (Fontes et al., 2022). According to Solarova et al. (2022), people are more likely to accept FRTs if they believe it is genuinely for security purposes and limited to that context. In private use, FRTs could make life easier, such as through online secure payments.

However, when surveillance enters the picture, FRT appears more beneficial for LEAs and providers than for the public. Moy (2021) posits that manual identification from camera footage is time-consuming and labour-intensive. In protests, governments may use FRT to monitor participants rapidly and efficiently, while companies may sell data to the government, potentially reinforcing governmental power if there is insufficient oversight and accountability (Zalnieriute, 2024). This phenomenon, known as surveillance capitalism (Zuboff, 2019), exploits human experience for commercial purposes, enabling companies to influence citizen and government behaviour, potentially eroding democratic principles and individual autonomy. Some practices show that FRT can assist law enforcement in efficiently identifying and apprehending suspects (Hamann & Rachel, 2019). However, evidence supporting this is limited, as others suggest FRTs could lead to misidentification and wrongful convictions, reinforcing inaccuracies associated with eyewitness identification (Moy, 2021). Governments and providers may adopt advanced technologies despite significant risks, not only to facilitate their duties but also to conduct surveillance on people. For example, the FBI’s FACE Services unit uses FRT to search driver’s licence databases, passport photos, and visa application photos, creating a biometric database of primarily law-abiding Americans without their knowledge (Garvie et al., 2016). While FRT offers potential for law enforcement, the absence of regulatory and practical evaluations reinforces existing risks, contributing to public distrust (Bragias et al., 2021). This raises questions about whether its use genuinely serves the public interest and whether the risks posed by surveillance technologies are worth taking.

3.2 Lower risks, better trust?

The relationship between risk, public trust, and public interest in the context of FRTs used by LEAs for public security is complex and debated. It is reasonable to assume that public acceptance of LEAs’ services would increase if risks are effectively mitigated and clearly communicated. However, this does not guarantee unanimous support, as there is no consensus that risk mitigation alone will enhance public trust (Matulionyte, 2024). Historically discriminated groups by law enforcement may remain sceptical, despite claims of public security benefits. Still, risk management is a tool entrusted by law with the purpose of evaluation of the suitability of risky implementations. The AIA’s risk management system may be designed with this goal in mind. The risks presented in this article focus on a selected few related to the processing of personal data with the aim of highlighting the importance of robust data governance frameworks.

4. FRT data governance in EU legislation

Algorithms begin and progress through their lifecycle by processing data, mostly personal data. While ongoing work is being done to develop techniques for creating well-performing algorithms with less data (Nguyen et al., 2023; Plangger et al., 2023), algorithms still require varying sizes and forms of data to function. Clearview AI is a FRT combined with a predictive policing tool developed by a private company and used by LEAs in several EU member states, including France, Belgium, and Italy. It holds approximately 30 billion images, mostly consisting of biometric data (Clayton and Derico, 2023). Biometric data is defined in the GDPR (Article 4, para.14), as the data generated through technical processes aimed at identifying or verifying the identity of individuals based on their physical, psychological, or behavioural characteristics. This definition distinguishes biometric data from other types of personal data by emphasising the importance of technical processes. In order for images to be classified as biometric data, they must be processed through a system that matches them with an individual’s identity (Article 9 GDPR). According to the definition, either the data processed in the FRT or the output resulting from that processing is considered personal data. For instance, personal data includes the output of combining an individual’s current psychological state, as revealed through facial images, with their past criminal record. The AIA follows the same approach as the GDPR in defining biometric data. This connection demonstrates that the risks associated with FRTs under the GDPR are also relevant to the AIA. These risks stem from compliance issues, legislation deficiencies, and practical implications that ultimately impact public opinion on the necessity of FRTs. By recognising and mitigating the risks outlined below, there is a greater chance of achieving a balance between public security and privacy by enhancing transparency.

4.1 Risk one: Violation of general principles – data minimisation challenge

The use of FRTs in public spaces has the potential to violate individuals’ rights to data protection which contradicts the general principles governing the processing of personal data outlined in the Article 5 of the GDPR, particularly the data minimisation rule. This rule has already been breached in practice, leading public authorities to intervene in terms of protecting public interest. The Italian DPA penalised Clearview AI’s FRT tool, as they violated rules related to transparency and purpose limitation by collecting biometric and geolocation data without a proper legal basis (EDPB, 2022). Despite facing fines exceeding €20 million from various EU data protection authorities (Noyb, 2023), Clearview AI and similar companies continue to develop algorithms using unrelated data.

The principle of data minimisation is not only related to the protection of individuals but also to the protection of the public in general (FRA, 2018, p. 125). LEAs’ wide range of data processing activities without explicit rules may cause fear of constant surveillance. As a result, pervasive surveillance significantly undermines individuals’ autonomy, making them feel compelled to conform to expected behaviours in public spaces (Norris, 2002). These spaces are characterised by their accessibility to the general public, regardless of whether they are publicly or privately owned. Public spaces are often referred to as “uncontrolled environments” (Consultative Committee, 2021, p.5), and the use of surveillance cameras means that individuals who pass by them become subjects of surveillance without any justification, leading to an environment controlled by authorities. Instead of freely expressing themselves, the knowledge of being watched forces people to conform to societal norms and expectations, thereby limiting their freedom and self-expression (Lyon, 2022). This is mainly because they often have limited knowledge about the fate of their data once it is captured by FRTs. Even if they were aware, they may face challenges in exercising their rights, such as the right to erasure under Article 17 of the GDPR or the right to restriction of processing of their data under Article 18 of the GDPR. When surveillance is targeted at a group, they have almost no chance of escaping the surveillance, and they cannot collectively request their rights since the GDPR does not recognise collective rights. In 2018, police used live FRT to scan the faces of individuals participating in a peaceful protest in a public space in the UK. This action led to a legal case as everyone’s face was scanned without being informed about possible algorithmic evaluation. As a result, the protestors felt unsafe when exercising their right to protest (Bridges v CCSWP, 2019). Despite the FRT deployed to enhance public security, its operation resulted in suppression of individuals, providing no tangible benefit.

4.2 Risk two: The challenge of limiting the purposes

The inherent nature of AI systems poses a significant challenge in terms of processing data, which diverges from the fundamental principle of processing personal data for a specific purpose (Article 5(b) GDPR). While biometric data may initially be collected for a specific purpose, AI algorithms can be trained to generate outputs that may suggest alternative purposes, such as using CCTVs installed in shopping centres for employee monitoring (EDPB, 2020) or identifying family connections and ethnicity, potentially leading to an Orwellian society. This functional shift, known as function creep (Koops, 2021), allows data to be processed beyond its primary purpose, which is permissible only under certain criteria, such as with the consent of the data subject (Article 6(1)(a) GDPR). However, consent practices can be deceptive, particularly when used by LEAs. For instance, Clearview AI evaluated data not only for system development purposes but also for surveillance without individuals’ consent (Rezende, 2020). This lack of transparency and individual agency underlines the power disparity between authorities and individuals, making it challenging for the public to assess risks or limit surveillance. Despite efforts by DPAs to intervene, such as the Swedish DPA penalising the Swedish police for using Clearview AI without specific purpose indication for surveillance (EDPB, 2021), technology evolves rapidly, outpacing regulatory responses to stop function creep. LEAs, as part of their public security responsibilities, are increasingly using advanced AI tools such as ChatGPT without obtaining individual consent and without informing the public (Europol, 2023). Additionally, techniques like soft biometry are being developed to assist in crime prediction based on a single piece of information, such as the colour of someone’s trousers (Wang et al., 2005) that could lead to identifying a person’s ethnicity. These developments have increased doubts about the use of FRTs, which are supposed to ensure public security, but not a wide range of surveillance.

Function creep is usually sought to be legalised by obtaining consent from individuals. This approach, combined with illegal consent practices, could lead to even wider problems, particularly if surveillance is targeted at specific groups. For example, a school in Sweden conducted a trial of FRT to monitor students’ attendance. However, this practice was deemed unlawful by the Swedish DPA as the data was processed without explicit consent. Two high schools in France tested a FRT to track students’ attendance, similar to the Swedish practice, and the implementation was found unlawful by the French DPA as consent was not properly obtained (CAIDP, 2020). Even though consent could help in achieving successful AI for social good (Floridi et al., 2020), people usually do not have the option to opt out of FRT evaluations in public spaces (Ada Lovelace Institute, 2019). In practice, an illusion to individuals is created as consent is the only legal basis for FRTs (Vogiatzoglou and Marquenie, 2022). An example is the FRT-based solution that was initially developed for pandemic-related purposes, and thus obtained consent for that specific objective, later employed for the identification of problem gamblers with the same data counting around 1 million records (Pearson, 2024). Apparently, the purpose which the consent was given is distinct from the initial one. Additionally, in the previous examples, students essentially pay for their (free) education with their biometric data. The reason for the school administration’s adoption of this implementation, whether for student surveillance or to aid in their educational progress, may be unclear to parents who were not involved in the decision-making process and did not request it. As such, they have raised concerns, and here the consent has at LEAs served to raise awareness among parents and correct the improper implementation.

4.3 Risk three: The data accuracy challenge

It is crucial to remember that the life-source of an AI system is data. In order for an AI system to generate accurate and unbiased outputs, Article 5(d) of the GDPR suggests that the data processed during the life cycle of an AI system must be accurate and up-to-date. In practice, this principle could help prevent individuals from having wrong accusations based on inaccurate data in police databases which may even result in imprisonment (Hill, 2020). Similarly, accurate data could prevent individuals from being repeatedly stopped and questioned by the police (Dimitrov-Kazakov v. Bulgaria 2011). These realities raise concerns about the reliability of the systems and whether they can ensure public security and justice. To mitigate the risk, one solution is to continuously update police databases with data from the operational field of FRT and improve the accuracy of the algorithm (Bergman et al., 2023). However, this potentially would reinforce surveillance in public (Raposo, 2022). Another solution could be referring to an aggregated database to update the system. This database could be open source, allowing for testable accuracy levels (Delikat, p. 61). However, in reality, providers may prefer to use their own closed database, where accuracy levels cannot be directly tested (Fábián & Gulyás 2021). For instance, the reliability of the system might be compromised if the database includes fake images that do not represent real individuals. It is important that both the LEA and the public can question the source of the data. Technical standards (ASTM, 2023) could help when training the algorithm for FRTs, considering various factors that can influence accuracy, such as hardware specifications, image angles, and data preparation processes. However, these details are often not transparently communicated to the public.

In legal documents and technical contexts, it is important to distinguish between two types of accuracy. The GDPR approaches the principle of data accuracy as the requirement for data to be correct, up-to-date, and rectifiable. In contrast, accuracy in AI terminology refers to the level of accuracy of the output generated by the algorithm and is mathematically or statistically presented. The GDPR refers not only to the accuracy of first-hand personal data but also to indirectly processed personal data that can identify individuals, such as rankings (Hacker et al., 2023). FRTs’ rankings should also be considered as personal data and carry specific risks related to the processing of personal data. Therefore, evaluations from FRTs are also considered personal data, as they probabilistically rank people according to their risk score. As will be demonstrated later, the AIA effectively bridges the gap between data accuracy and system accuracy, although challenges still persist.

4.4 Risk four: Administrative challenges

In the public sector, software and hardware products are typically outsourced to the private IT sector, which offers a wide range of high-quality systems that a LEA cannot develop in-house. According to the USGAO report on the algorithms used in federal law enforcement (2020), half of the AI-based systems used by US federal LEAs are outsourced to private companies. The report highlights that authorities may lack technical knowledge of the FRTs they are using, which could hinder their ability to anticipate and assess the human rights risks posed by these systems. Additionally, a separate study found that most Americans would welcome FRTs if police were properly trained in their use (Raine et al., 2022). However, in practice, the police tend to prioritise the effectiveness of the software over its development and operation (Urqhuart, p.15). The LEAs should ensure that they clearly communicate their roles and duties in operating FRTs to the public, and participate in possible technical settings of the system (see the system accuracy title later for an in-depth analysis).

Identifying clear roles in the operation of FRT is a tool to enhance transparency and accountability. Once faced with an algorithmic evaluation, people often are confused on whom to turn to ask for explanations specific to their cases. The GDPR identifies three actors involved with data handling that are data controllers, data processors and joint controllers. Their responsibility (either sole, joint, or shared) level is not equally weighted between these categories since they take different roles. It is common practice in the IT sector that the data passes back and forth between many data controllers and processors, which may make it difficult to identify the real responsibility (Gültekin-Várkonyi, 2020, p. 198). Even from the legal perspective, distinguishing and identifying these roles clearly is uncertain, which reduces the impact of the fines to be imposed (Mahieu et al., 2019). Moreover, controlling processed data proves difficult as it neither falls under public domain nor is readily accessible to public institutions. The commercial development of FRTs by private companies further exacerbates the situation, as public trust in these entities tends to be lower than in public actors (Ada Lovelace Institute et al., 2021). The use of ChatGPT increases among LEAs, they may sometimes unknowingly transmit personal data to other partners of OpenAI, potentially eroding public trust if such usage remains undisclosed. Hence, the importance of defining clear roles cannot be overstated, as the deployment of FRT systems by unprepared law enforcement personnel risks eroding public trust without proper understanding and oversight.

4.5 Summary

The table below summarises the mentioned risks associated with the use of FRTs processing personal data in public spaces evaluated under the GDPR.

The first risk involves violating the data minimisation rule, which can occur both technically and practically. This violation can lead to excessive surveillance and a loss of individual autonomy referring to having legal consequences. The second risk, function creep, arises from the technical and practical misuse of data for purposes beyond its original intent, which undermines legal rules relevant to transparency and legitimate data processing. The third risk addresses data accuracy issues that are technical and legal, which can lead to wrongful accusations and diminish the reliability of AI systems. The fourth risk focuses on administrative challenges that stem from practical and legal uncertainties, particularly the outsourcing of AI systems, which hampers effective oversight and accountability. Together, these risks regarding processing personal data in FRTs illustrate how surveillance technologies, while intended to enhance public security, can often compromise public interest by violating the right to data protection.

5. Evaluation of the four risks under the AIA

This section analyses the risks introduced by FRTs in law enforcement, focusing on the viewpoint of the AIA and aligning with the scrutiny applied in the GDPR analysis. The AIA classifies AI systems into four risk levels: those with unacceptable risk, high risks that are acceptable under certain regulatory conditions, limited risk, and minimal risk. FRTs that are classified as unacceptable risk should be banned and only allowed in limited, clear, and strict exceptional cases. The AIA establishes general principles aimed at upholding fundamental rights by LEAs and establishing minimum standards for AI systems, but these principles may lack the requisite specificity for FRTs in certain aspects, potentially undermining its main purpose for operation.

5.1 Risk one: Violation of general principles-definitional challenges

The AIA adds further complexities to those mentioned in the GDPR section when interpreting rules and principles. The definitional issues in the Act make it even more challenging to demonstrate transparency of the FRT to the public, thereby reinforcing existing distrust. The very first issue relates to the definition of FRT from both a legal and technical perspective. Finklea et al. (2020) states that FRT, which may seem like a simple term, can have different technical and legal meanings, and that (like most of the technological terms) there is no clear definition of this technology. The term as it appears in the AIA still does not fully encompass all possible types of AI and remains narrowly defined. For instance, the definition of GPT which could also process biometric data now remains overly inclusive in the Act, attempting to simplify the technology underestimating its technical capabilities (Hacker et al. 2023). This allows providers to interpret the law and ethical standards in a personalised manner, which may differ from the intended meaning (Hacker et al. 2023).

A significant definitional shortcoming in the AIA pertains to the definition of biometric data. The AIA references the GDPR and the LED for defining biometric data, which has a special relation to facial recognition only under French data protection law (Christakis et al., 2022a). This term requires a more detailed evaluation, particularly concerning its application in FRTs used in public spaces for security purposes, and must be clearly explained to the public. Further, the AIA introduces several terms in Article 3, including “biometric identification” (para. 35), “biometric verification” (para. 36), “biometric categorisation system” (para. 40), “remote biometric identification system” (para. 41), “‘real-time’ remote biometric identification” (para. 42), and “post-remote biometric identification system” (para. 43). These terms may overlap or lack clarity from both practical and technical perspectives, creating a “terms salad with no taste”, which only complicates understanding for both deployers and providers. This terminological plurality can impair the public’s comprehension of how these technologies function. Moreover, law enforcement agencies (LEAs) often withhold detailed information about the tools they use, citing security reasons. This tendency towards secrecy exacerbates the problem of asymmetric information, where LEAs possess more knowledge than the public. Such an imbalance can result in the deployment of these technologies without adequate public scrutiny, justified under the pretext of security threats (Hacker, 2018, p. 90). Consequently, the actual impact of FRT on public security remains unclear, potentially eroding public trust in these technologies.

The final challenge to be presented here is relevant to the identification of the law applicable to FRTs. Already, there is no legislation clearly limiting the use of technologies by LEAs (Christakis et al., 2021). Existing legislation governing technology use by LEAs is either lacking or opaque, hindering individuals’ ability to obtain information and assert their rights (Sherwin et al., 2019; Li, 2019) and weakening their position under power. The challenge of categorising technologies under appropriate legislation is evident, as various AI tools used by law enforcement fall under different laws, lacking specificity to address key issues (Karsai, 2020). Applicable ones allows MS to set rules on biometric data processing, leading to varied interpretations and applications across the EU, weakening the position of the public authorities in defending public interest. Assessing the use of GPT tools by LEA under the GDPR could typically result in a blanket ban due to their capabilities for real-time voice and image analysis, which involve processing biometric data without public awareness. For instance, the Italian DPA temporarily banned ChatGPT for these reasons (Helleputte et al., 2024), although its use remains permitted. On the other hand, the Austrian data protection authority’s leniency towards Clearview AI exemplifies divergent regulatory approaches (TIPIK Legal, 2021). This regulatory and definitional ambiguity underscores the need for a clear and uniform approach in application to protect public interest and individual rights.

5.2 Risk two: The challenge of limiting the purposes

The AIA introduces the term intended purpose (Article 3(12)) to align closely with the GDPR, emphasising that AI systems are designed for specific purposes as defined by the provider. Any deviation from this intended purpose is considered misuse. However, if the primary purpose is not clearly adhered to by the deployer, public security could be compromised by the reuse of data – often legally permitted but not always aligned with the intended purpose. For example, in Hungary, facial image systems are primarily used for identification purposes and are regulated in detail (Gárdonyi, 2020). However, the use of facial image data by legally authorised LEAs is complicated by the fact that the law itself (for example, Article 6 of Act CLXXXVIII, 2015) identifies a non-exhaustive list of areas in which FRTs can operate, albeit under certain conditions (Act CLXXXVIII, 2015, p.27). Such reuse, although not technically misuse, may still challenge public security interests if the AI system’s intended purpose evolves during use.

The AIA explicitly prohibits real-time remote biometric identification systems, with some exceptions (Article 5(1)(h)). In the parliamentary draft of the AIA (EP, 14 June 2023, T9-0236/2023), the EP advocated for extending this prohibition to include post-remote identification systems. The proposed amendment (nr. 227) in Article 5(1), with the introduction of a new point (d), suggested a ban on the use of post-remote identification systems by LEAs in publicly accessible areas, except with pre-judicial authorization for targeted searches related to specific serious crimes, as defined in Article 83(1) of the Treaty on the Functioning of the European Union. This amendment would have potentially set a higher barrier for the implementation of post-biometric identification systems. However, this proposal was not adopted in the final text, which maintained the prohibition specifically for real-time biometric identification systems. Consequently, post-remote identification systems are classified not as prohibited AI systems but as high-risk AI systems, with broader exceptions for their use. Additionally, the proposed amendment initially required pre-judicial authorization, but the final version replaced this with administrative authorization, thereby weakening judicial protection of the public interest before LEAs. This change raises concerns about the potential for governmental authorities to repurpose data in post-remote systems for broader uses, leading to inconsistencies in the application of the AIA and varying levels of protection for European citizens. At the time of writing, a joint paper by Germany, France, and Italy emphasised that the AIA should be applied to avoid stifling innovation rather than imposing sanctions (Reuters, 2023; Gstrein et al., 2024). Although all EU member states eventually agreed on the final text, this joint action indicates that there were divergent interests among the members and EU decision-makers regarding the AIA, which could also manifest in the application of security-related exceptions for public interest.

It should be noted that any method of identification, whether remote or not, post or real-time, could lead to function creep and surveillance as the data is used in the system. When the system is fed back by the data collected either in real time post, “threats are not reduced just because authorities or companies have more time to review footage” (Jakubowska, 2023). For these reasons, the limitation of purpose and clear identification of intended purposes have a crucial impact on constraining the potential extensive surveillance practices of LEAs. According to a study (Delikat, 2021), the specified purposes for using FRTs in EU LEAs are generally broad and vary among them, particularly in the context of surveillance. In addition, LEAs often delegate decisions about the fate of data in the system, such as when to store, delete and update personal data, to providers who may not have a full understanding of the legal requirements compared to the LEAs themselves. This dynamic prompts member states to implement measures that restrict data subjects’ right of access, ensuring the efficiency of police work while limiting the fundamental rights (FRA, 2019).

5.3 Risk three: The system accuracy challenge

The AIA smoothly makes the connection between data accuracy and system accuracy in the concept of law enforcement. The AIA’s Recital 67 highlights the training data to be high quality to ensure accuracy and avoid individuals in a discriminatory, incorrect, or unfair accusations. To ensure the success of a system that also serves public interest, this link is crucial. Article 15 of the AIA explicitly refers to the importance of accuracy in high-risk AI systems, which is considered one of the most crucial transparency requirements for such systems. To enhance accuracy, the expected level of precision and its metrics should be disclosed to deployers (not individuals or the public directly) as instructions. However, evaluating the performance of a FRT system involves different metrics, each providing a unique perspective on the system’s results as it is also the case for the FRT (EDPB, 2022). It is unrealistic to expect that all metrics will be included in a single document and easily understood by all LEAs without further communication. This raises several questions.

The initial question that arises is: who determines the planned and tested accuracy level in FRTs. Article 13(3)(2)(ii) AIA states that the accuracy level must be designed by the provider to complement transparency enabling deployers to (only) interpret outputs, without referring clearly to the possibility of the deployer to give any feedback on the planned or enabled accuracy level. It is clear that the provider alone decides on the relevant accuracy of the system during the system design phase. During the implementation of FRTs, there is a risk that the focus might shift solely to system performance metrics, potentially prioritising system performance over public interest considerations. For example, the Metropolitan Police discovered that over 98% of matches in their live biometric data processing during implementation were all in false positives (Big Brother Watch, 2018, p. 25). Different accuracy settings are required for FRTs used for border checks compared to criminal matches, as both applications directly affect fundamental rights. Even within the same context, achieving an acceptable level of accuracy demands a careful and detailed analysis. Distinguishing between classifying someone with 90% confidence as a potential criminal versus 60% confidence is crucial, as the difference significantly impacts an individual’s rights. It is evident that there are various approaches to determining the appropriate accuracy levels. The prospect of establishing a unified methodology at the EU level appears, given that the AIA involves the Commission, in collaboration with relevant stakeholders, in addressing the technical aspects of methodologies for measuring suitable accuracy levels in Article 15(2) AIA. Recital 74 of the AIA further specifies the Commission’s role in this task, drawing parallels to its involvement in Directive 2014/31/EU on non-automatic weighing instruments and Directive 2014/32/EU on measuring instruments. This points to a probability for the EU to take regulatory or non-regulatory actions to ensure AI accuracy in alignment with broader legal metrology principles.

This involvement raises two key questions. First, how can methodologies for general accuracy measures for a specific technology like FRT and its specific law enforcement applications be identified? If specificity is not sought, then why are the accuracy rules, including quality management rules specified in Article 17 of the AIA, not sufficient? The accuracy rules and the quality management system already give guidance on how the accuracy levels should be aligned with general principles. General methodologies for identifying accuracy levels in a specific technology may not work due to several factors. For instance, from a technical standpoint, face recognition is less accurate than other biometric data, such as fingerprinting (Garvie et al., 2016). Additionally, the potential for inaccuracy in these learning systems is high, as accuracy may change over time with new data. Consequently, periodic revisions on the specific accuracy levels, together with their methodologies, are necessary to ensure the system’s reliability. The AIA lacks flexibility and clarity in this regard. The interpretation of relevant articles (Article 15 (2) AIA) should have indicated that accuracy levels are not fixed measurements and should be adjustable based on usage scenarios. Since an FRT cannot be developed solely for one purpose (see “Purpose limitation” section), its accuracy level cannot be singular.

Second, when and how will the Commission, as a political entity, reach a consensus on methodologies for accuracy measurement tools for AI systems used by member states’ LEAs? Most parts of the Act will be effective from August 2026, less than two years away, and discussions on sensitive topics like measurement methodologies for high-risk AI systems used for public security have not yet taken place. It will be interesting to observe LEAs’ involvement in these discussions, representing their national interests, and whether they can find common ground beyond the general agreements already established.

The ideal solution could have been referred to a more local level, without involving the political bodies in the process. For example, the rules could guide the provider to provide the deployer with detailed explanations of the intended levels of accuracy. Feedback from the provider could be required, which would then increase the transparency of the system (Hacker & Passoth, 2022). In this process, the provider should be aware of the modelling steps needed to mitigate any accuracy risks arising from incorrect use cases. This is because a technology designed for verification might not be effective for searching a criminal database. Further, to avoid malfunctioning and erroneous outputs, it is essential to use not only data but also highly representative data. As Fontes et al. (2022) explained in their analysis of a privacy-focused Corona app, which dropped representativeness by not collecting sensitive information, highly representative data must be fed to the systems to avoid incorrect outputs. LEAs, who are likely well aware of the groups that might be underrepresented in FRTs, could provide relevant instructions to the providers. LEAs should guide the provider to understand each specific use case and the acceptable accuracy levels from both technical and legal perspectives. This approach enables the provider to gain an understanding of the legal and contextual background of the FRT they develop in a timely manner, rather than at a later stage. Furthermore, it allows the LEAs to communicate this understanding to the public. If the terminology used in technical documents could be translated into accessible language, it is probable that the public would provide feedback on such complex procedures if a feedback system were to be established. By collaborating with the public on such sensitive matters, it may be possible to diminish scepticism towards the systems (and their accuracy) (Urquhart & Melinda, 2021, p. 5), thereby fostering the legitimacy of their use.

5.4 Risk four: Administrative challenges

How difficult is it for individuals to identify a responsible person or persons behind FRTs operated in public spaces? Answering this question, similar to the discussion presented under the GDPR, is not an easy task for the public, and one of the reasons for such difficulty is the administrative relationship between the deployers and the providers. Private entities often exploit legal loopholes to avoid direct responsibility while still maximising profits. This does not imply that they are any less responsible than LEAs, both to LEAs and to the public, when their system is being used. FRT deployers should be able to assign the sole or partial responsibility to the providers, e.g. in case of systemic failure or in extensive data collection measures. The AIA repeats the complexity to administer the issue as per discussion of the same topic under the GDPR. According to Article 3 (46) of the AIA, law enforcement refers to “activities carried out by law enforcement authorities or on their behalf” for several purposes, such as the prevention of threats to public security. Based on this definition, private entities and even individuals could also act on behalf of law enforcement.

This arrangement often lacks clarity regarding the relationship between these external entities and LEAs, leaving the public uncertain about who should be held accountable for any misuse or errors, whether it be the LEA, the third-party operator, or both. It complicates for the LEA the issue of responsibility not only in cases of system failure (Moss & Metcalf, 2019), but raises uncertainties to the public about where to direct complaints or seek redress for potential data misuse, the LEA or the private entity? Moreover, the involvement of third-party operators, particularly those across different jurisdictions (such as the Clearview AI operated by a US-based company), may cause legal complexities. Varying standards for data protection, accuracy, and liability across jurisdictions can lead to difficulties in coordination and may result in legal conflicts or challenges in enforcing EU regulations in a wider sense.

5.5 Summary

The table below is to summarise the mentioned risks associated with the use of FRTs in public spaces evaluated under the AIA.

The first risk involves definitional issues which stems from legal challenges that create confusion about the technology’s scope and application, complicating effective regulation and public understanding. The second risk addresses the challenge of limiting FRT purposes which occurs from legal, technical, and practical reasons, where unclear or evolving uses can lead to misuse and excessive surveillance, potentially compromising public security. The third risk pertains to system accuracy, both a legal and a technical challenge, where inconsistent or inadequate accuracy standards can result in unreliable technology, impacting its effectiveness in ensuring security. Lastly, as a legal and practical challenge, administrative challenges arise from the complexities in assigning responsibility and ensuring accountability between LEAs and third-party providers, complicating oversight and enforcement. These risks collectively highlight how the deployment and regulation of FRTs can affect public security and interests, underscoring the need for clear, consistent guidelines and robust management to prevent misuse and ensure effective oversight.

6. A suggested solution: Enhancing the role of data protection impact assessment for a more legitimate FRT

In order to serve the public interest in a genuine manner, LEAs must demonstrate that FRTs benefit not only their operational needs but also the public good. A bottom-up approach, whereby LEAs gain public trust by first addressing individual privacy and data protection concerns, may prove more effective than a top-down legislative approach. A strategy based on incremental steps could help to avoid a total breakdown. One starting point with this aim is, as this article proposes, to enhance the Data Protection Impact Assessment (DPIA) and integrate it to the existing risk management system of the AIA, even if it is not a comprehensive solution alone.

The AIA introduces minimum standards to ensure that fundamental rights are respected throughout the lifecycle of AI systems. To ensure this, it introduces obligations for those involved in the development, marketing, and operation of the system. These obligations have much in common with the GDPR. For example, high-risk AI systems must undergo a fundamental rights impact assessment (FRIA) as indicated in the Article 27 of the AIA before they are placed on the market. This assessment is made by the deployer to measure in what ways a certain AI system would put fundamental rights at risk, from the aspects of defining how and when the system will be used, who it might affect, and how humans will oversee the system. In the end, the deployers also should present how these risks would be mitigated. This is similar to the DPIA required by Article 35 of the GDPR. The law only requires the FRIA for aspects not covered by the DPIA, which creates a link between the two and they complement each other.

It should be emphasised that the DPIA focuses on assessing risks related to data processing but does not cover potential issues such as government surveillance, bias, or accuracy testing. The DPIA is conducted by data controllers to meet accountability and transparency requirements primarily for public authorities, not directly for the public. To address the broader risks highlighted in this article, it is proposed to expand the scope of the DPIA by incorporating new assessment topics, rather than creating entirely new frameworks. This approach allows for specific applications, such as in healthcare, without redundancy (Gültekin-Várkonyi & Gradišek, 2020). In contrast, the FRIA is carried out by the deployers of AI systems, often without the involvement of providers who could offer essential technical guidance. For instance, evaluating the risks posed by technologies like FRTs, especially those affecting specific groups (as indicated in Article 27(1)(d)), would benefit from collaboration with the original system designers. However, the implementation of these assessments might be less problematic under the AIA, which introduces a comprehensive risk management system. This system is designed to be continuous and iterative, covering the entire lifecycle of the AI system. It integrates the principles of both DPIA and FRIA, ensuring that risks to fundamental rights and data protection are continuously identified, evaluated, and mitigated. The goal is to ensure that AI systems are developed and deployed in ways that protect individual rights and serve the public interest.

At this point, DPIAs could be imagined as an integrated part of a broader risk management system, but it still should be considered as a separate and specific mechanism for evaluation of the personal data to be processed. For LEAs, DPIAs offer a unique perspective that allows them to fully assess risks through the lens of public perception specific to their personal data protection, thereby facilitating more informed decision-making processes. Including public engagement in DPIAs could promote transparency and increase confidence in the intended function of LEAs, which is to provide security rather than facilitate widespread surveillance. However, the existing literature and practices do not refer to a specific DPIA for FRTs at LEAs during the course of my research for this article. The only literature found during the preparation of this article was by Castellucia and Le Mateyer (2020), who proposed a DPIA specifically designed for FRTs. It is important to note that this report was published before the AIA and therefore does not take it into account.

Existing literature on DPIA strongly emphasises the need for public participation in impact assessments (Bondi et al., 2021). Public consultation is often overlooked in the formulation of DPIAs, and existing practices often fail to inform the public effectively, relying solely on ex-post public awareness initiatives, such as publishing information on police websites or social media accounts, as exemplified by the 2020 Bridges vs CCSWP case. If an assessment of data processing is to be made, it must be made before data is processed in an AI system, and should include public access and public consultation before data is processed (Moss et al., 2021). If the various interests of the public (data protection interests, as specifically focused on this article) are not represented during the development of the system and if they are not given the opportunity to contribute, this can lead to an incomplete system design, which can later cause the project to fail (Züger & Asghari, 2023, p. 820). This failure would affect both providers and LEAs, not only from a legal point of view, but also from a technical and practical point of view. Sloane et al. (2022) suggest several ways in which stakeholders could participate in the DPIA. For example, FRT providers could consider a consultative approach to participation in conjunction with the LEAs, as mentioned in the system accuracy section of this article.

In light of the findings from the literature presented, it is paramount to ensure the public’s active involvement and representation in the design process of FRTs. Individuals and the public could be involved in DPIAs in many ways. For example, providers and deployers could consider involving members of the public as external reviewers of the DPIA. This way, public feedback should be taken into account in the further decision-making process. To ensure impartiality, but also to make participation attractive, they could offer benefits to participants, such as a gift (e.g. a cinema ticket), which would not bias their feedback towards the interests of the deployer or the provider. In addition, academics and non-profit organisations could be involved in the preparation of the DPIA. Based on all of these analyses, I suggest in the table below that several questions be considered to be raised during the DPIA, which represent sub-assessment points for the already existing requirements specified in both pieces of legislation.

The table presents the points discussed in the previous sections under the four risk headings. As a reminder, each title has addressed the risks from either a technical, legal, or practical perspective. Therefore, the proposed questions to extend the scope of the DPIA to specifically focus on the use of FRT in LEAs also contain practical, legal and technical assessment points that aim to guide these actors to operate FRT in a more data protection rights friendly way. A note must be made here about the language to be used in the DPIAs, especially since they are supposed to be made publicly available. There is a difference between the purpose of the FRT (authentication, identification, or surveillance) and the information on these purposes to be provided to the public (Matulionyte, 2024). As DPIA is also one of the transparency tools and, since it contains many technical and legal assessments, one DPIA format designed in a general language might not be enough. The information to be delivered in the DPIA should be selected based on the public knowledge of the FRT, taking into account the potential for there to be individuals with limited expertise in FRTs and another group of the public who are experts on the topic. Finally, ensuring the delivery of communication in languages other than the dominant language is crucial for fostering inclusiveness.

This table is not final and is indeed open to further improvement. The questions in the relevant areas aim to fill current gaps that may arise in practice and can be further improved and are therefore complementary to those that already exist in the legal requirements. In an era of limitless technology, aligning legal rules with practical guidance may help strengthen the protection of human rights in the face of new and relatively risky technology.

7. Conclusion

FRT offers LEAs significant opportunities in public security, with applications in authentication, identification, and profiling. As these technologies heavily rely on biometric data, their use by law enforcement is scrutinised only under legislation, mainly the GDPR and AIA, emphasising the need for more practices to comply with more public interest centric applications.

The risks posed by FRTs in law enforcement include challenges with data minimisation, function creep, accuracy, and administrative complexities. Data minimisation concerns arise from the constant collection of biometric data, potentially infringing on privacy and anonymity. Function creep occurs when algorithms exceed their intended purposes, leading to potential misuse by both private entities and LEAs. Accuracy issues under the GDPR and AIA pose difficulties in correcting and updating data, while the AIA’s allocation of responsibility adds to these challenges. Administrative complications stem from outsourcing FRTs, creating a complex network of accountability that could undermine public trust.

To mitigate these risks and enhance public confidence, LEAs must clearly demonstrate that the tool is planned to be used for its sole purpose. Extending the scope of the DPIA as an integral part of FRIA to specifically address FRT in law enforcement could improve transparency and effectiveness. Ultimately, the development and use of FRTs should be guided by public opinion to ensure they meet both security needs and individual privacy concerns.