Читать книгу Methodologies and Challenges in Forensic Linguistic Casework - Группа авторов - Страница 24

Since Dror et al.’s (2005, 2006) work gave the issue prominence, confirmation bias in forensic evidence has received considerable attention. Dror’s work on fingerprint examiners (e.g., 2005) and even forensic DNA analysis (2011) demonstrates how the conclusions of honest, competent forensic scientists are subject to cognitive bias and will be influenced by the briefings they are given prior to carrying out their examinations. Dror’s work raises issues as to how such bias can be designed out of forensic practice. The UK Forensic Regulator’s (2015) guidance paper on cognitive bias in forensic evidence examinations discusses the risks and provides methods for risk reduction as well as specific examples across the forensic sciences. For example, with regard to fingerprint examination, the regulator suggests that processes should be examined and adapted to “remove or limit contextual information that adds no tangible value to the fingerprint examination process.” (Forensic Regulator, 2015, 8.5.3b).

With regard to discussion of the Starbuck case, the question this chapter addresses is the extent to which we can design processes in forensic authorship analysis that address issues of cognitive bias. We do this through the presentation of a specific case – the murder in 2010 of Debbie Starbuck – and of our role in that investigation, and we describe how we attempted to address potential biases through the procedures we adopted and then evaluate the effectiveness of those attempts.

Forensic authorship analysis is a broad discipline, but the main tasks can be characterized as comparative authorship analysis, where the aim is to provide evidence that helps attribute one or more texts to a particular author by comparison with writings of undisputed authorship (as carried out in the Starbuck case), and author profiling, where the aim is to identify the sociolinguistic background of the writer of a text. (See Grant & MacLeod, 2020, Ch. 6, for a discussion of these and other possible authorship analysis tasks.) Whatever the task, authorship analysis is a lot less secure against cognitive biases than many other forms of forensic examination, and, before turning to the specifics of the case, we examine here why this might be the case.

First, forensic authorship analysis is largely still the domain of individual experts, working from universities or small consultancies, which typically contain just one or two analysts. From cases reported in the literature, most analyses are carried out by individuals, not as part of an established institutional procedure with quality controls, such as verification of reports and opinions by a second expert. The individualistic nature of authorship analyses leaves them open to potential cognitive biases.

In terms of approach, there is still some division between analysts who adopt wholly computational approaches to authorship questions, which are sometimes referred to as stylometric approaches, and approaches that rely more on linguistic training and expertise in text analysis, which can be referred to as stylistic approaches. The vulnerability to bias remains whether the analysis is largely computational or depends on an individual’s skills in noticing or eliciting style features that might be indicative of the style of author A or author B, although the nature of this bias can change.

For the stylistic analyst, the risk of bias may be more obvious: unconscious bias can affect decisions to include or exclude a particular feature, and indeed noticing or not noticing a particular feature may be a source of bias. However, it is also the case that in stylometric analyses a series of design decisions also need to be made—biases can creep in with regard to the selection of comparison materials, the identification of features to be elicited, and the statistical or other methods of prediction applied. Argamon (2018) usefully discusses the various possible decision points and pitfalls for computational forensic authorship analysis, and every decision point is also a point at which conscious or unconscious bias can enter an analysis.

Second, unlike with the analysis of a fingerprint or a DNA sample, in authorship analyses it is often not possible to isolate the sampled material from the story of the investigation. In fingerprint examination, it may be possible, as advised by the UK Forensic Regulator, to avoid telling an examiner much of the contextual information about a crime, but in linguistic analysis it is often the case that the texts themselves contain part of the wider story of the investigation. This unavoidable knowledge of contextual information clearly gives rise to potential bias.

Third, in authorship analysis, there is strong acknowledgment that cross-genre and the various interpersonal sources of linguistic variation give rise to considerable uncertainty around any conclusion of authorship. The issue of selection of materials of known authorship—either as direct comparison materials or as background materials to determine a base rate within a particular genre, context, or community of practice—is one of the most crucial decisions to be made in any authorship analysis. It is also a decision that is one of the most significant in producing an erroneous or biased result.

Opinion and judgment are always involved in assessing comparison material and in drawing a conclusion that the specific texts constitute an adequate comparison set in any particular problem. A bad choice of comparison corpus might mean that an analyst is led to believe that a particular feature is distinctive and associated with a particular author, when in fact the feature arises due to variation in register, genre, or a community of practice.²

For example, if an the analyst noticed that a particular author used “because” as a preposition as opposed to a subordinating conjunction in their business emails, then they might want to investigate how distinctive this is. In such a case what would make a good comparison corpus? A contemporary corpus of writings in computer-mediated communications might be available, or there may only be a slightly older corpus of business emails. The former might show that recently this is now fairly common usage across a range of computer-mediated communications such as social media and blog posts, and the latter may demonstrate it was rare for most authors writing business emails (but a few years ago).

The risk of error here is that the analyst comes to believe that the feature is an authorship marker, when in fact it indicates the queried text is of the register or the community of practice from which it was drawn. This is what in social science is referred to as a validity issue (see Grant & Baker, 2001).³ The risk of confirmation bias is that a comparison corpus is selected to best support the analyst’s preconceived hypothesis, and this can be exacerbated by the time and resource available to investigate a particular case. Examination of emails from the individual’s company might show, for example, that “because-as-preposition” is locally common within emails from that particular business, but it would take considerable effort to discover this.

In considering the sources of these biases more generally, we have to recognize the internal and external psychological pressures brought to bear on the analyst. These range from a natural desire to want to help through to broader concerns about building a reputation or a business. These pressures create potential bias in the decision-making process even before any client hypothesis has been heard, particularly where the decisions are a bit more balanced or nuanced. In any authorship analysis, significant decisions about the design of the analysis are very real and will affect the result. Very often, the best decision—the most expert judgment—is that no analysis is possible with the provided material.

One of the most fundamental decisions to be made concerns how much information an analyst should know about a case. There is an important tension between knowing as little contextual information as possible to avoid bias but knowing as much information as possible about the texts to ensure that there is a basis for a sound comparison in the texts analyzed. These judgments of text selection and of which type of analysis to apply are at the heart of expertise in forensic authorship analysis and are not mitigated by taking a wholly computational approach. This issue and all of the sources of bias need proper consideration, and, where possible, the biases need to be mitigated or designed out of the analyses. In turning to the Starbuck case, we show how we attempted this balance, reflect on the methods applied in this case, and make recommendations for future developments in this area.