Читать книгу Empire of Secrets: British Intelligence, the Cold War and the Twilight of Empire - Calder Walton - Страница 11

One advantage of the secret service is that it has no worrying audit. The service is ludicrously starved, of course, but the funds are administered by a few men who do not call for vouchers or present itemised accounts …

He considered the years to come when Kim would have been entered and made to the Great Game that never ceases day and night, throughout India.

RUDYARD KIPLING, Kim¹

Governments have conducted espionage and intelligence-gathering efforts for centuries. Indeed, intelligence-gathering – often said to be the world’s second oldest profession – is as old as governments themselves. In Britain, there was a ‘secret service’ operating at least since the reign of Queen Elizabeth I in the sixteenth century, which under Sir Francis Walsingham was tasked to gather intelligence on the Spanish Armada and to uncover various Catholic intrigues and plots. However, it was not until the nineteenth century, and more importantly the early twentieth century, that the British government began to devote significant resources to intelligence, and turn it into a professional, bureaucratic enterprise. Despite Britain’s long history of clandestine espionage work, in fact it was not in the ‘domestic’ realm that its intelligence-gathering was to develop most rapidly. Instead, it was in the British empire, which in the course of the nineteenth and twentieth centuries grew to become the greatest empire in world history, that intelligence found a particularly important role.²

From the earliest days of the British intelligence community, which was established in the early twentieth century, there was a close connection between intelligence-gathering and empire. It is not an exaggeration to say that in its early years British intelligence was British imperial intelligence.This is not surprising when it is considered that intelligence played an essential role in the administration of the empire, which by the 1920s had grown to encompass one-quarter of the world’s territory and population. After 1918, as one geographer proudly commented, the empire reached its widest extent, covering ‘one continent, a hundred peninsulas, five hundred promontories, a thousand lakes, two thousand rivers, ten thousand islands’. The empire had four kinds of dependent territories: colonies, protectorates, protected states and trust territories. At one end of the spectrum, colonies were those territories, like Kenya, where the monarch of the United Kingdom had absolute sovereignty, while trust territories, at the other end of the spectrum, were those assigned to Great Britain for administration under a special mandate, like Palestine. There was often little practical difference between colonies and protectorates. The Colonial Office usually referred to territories under ‘traditional’ rulers, with a British resident, as ‘protected states’. The typology of these dependent states was incredibly confusing (sometimes even to the Colonial Office itself).

One reason for the importance of intelligence in the empire was the lack of sheer manpower required to cover such enormous territories. Even at its height, British rule in India was maintained through an incredibly small number of administrative officials, with the renowned Indian Civil Service in the Raj boasting a total of just 1,200 posts, at a time when the population of India was probably around 280 million. Before 1939 the Indian army of 200,000 men, together with a British garrison of 60,000, was responsible for keeping the peace on land from Egypt to Hong Kong – British territories ‘East of Suez’, to use the phrase from the time. With such meagre resources at its disposal, British rule in India required up-to-date and reliable information on its enemies, both imagined and real. This was acquired through networks of informants and agents, and from intercepted communications. It is little wonder that, as one study has termed it, the British empire in the nineteenth century was an ‘empire of information’.³

Intelligence-gathering also came to the forefront in Britain’s imperial military campaigns in the late nineteenth and early twentieth centuries. One of the most exhilarating theatres for intelligence operations, or spying, lay in India’s North-West Frontier – now the tribal borderlands of Pakistan – where Victorian Britain fought the ‘Great Game’ with Russia, a conflict memorably portrayed by Rudyard Kipling in Kim, arguably one of the greatest espionage novels of all time. In Kim, Kipling described the ‘Great Game’ as essentially an intelligence conflict, which ‘never ceases day or night’, with both Britain and Russia running spies and informants to discover the other’s intentions. However, the reality was that it was often not difficult for Russia to spot British imperial intelligence agents: they were often extremely amateurish and deployed flimsy covers, variously posing as butterfly collectors, archaeologists and ethnographers. That said, it was in the ‘Great Game’ that some distinctly more professional forms of intelligence-gathering were born, particularly in a process that would later become known as signals intelligence (SIGINT), the interception and decryption of communications, or ‘signals’. In 1844 the Indian army pioneered one of the first permanent code-breaking bureaus in the world, which gained notable successes in reading Russian communications long before any similar European SIGINT agency had done so. The British military also made innovative use of intelligence during its campaigns in Egypt in the 1880s, successfully deploying a series of agents and scouts to reconnoitre the location of Egyptian forces in the desert.⁴

The very process of Britain’s colonial expansion in the Victorian period, especially during the so-called ‘scramble for Africa’ beginning in the 1880s, necessitated new forms of systematic intelligence-gathering, such as mapping and census-taking. In undertaking such activities, Britain was not acting differently from its imperial rivals at the time – France, Russia, Germany and Italy. Before any colonial power could dominate, control and exploit colonial populations, in Africa or elsewhere, it first had to map them. In practice, however, the process of mapping an empire often ignored its realities. Maps imposed European geometrical patterns on amorphous landscapes, drawing frontiers that cut through tribal communities as well as ethnic and linguistic groups. To this day, it is not difficult to spot the borders of those countries, particularly in Africa, which were drawn by European cartographers: many are arranged at right angles and slice through geographical features and ethnographic groupings. Sometimes European powers displaced and resettled colonial populations in order to make them reflect the ethnographic colonial maps. In the ‘white man’s burden’ of colonial rule, subtle realities did not matter.⁵

Given all that, it is no coincidence that Britain’s first Directorate of Military Intelligence, established in 1887, grew from the Topographical and Statistical Department in the War Office, which was responsible for mapping much of the British empire. Moreover, it was a violent colonial ‘small war’ in an outpost of the British empire, the Second Anglo-Boer War in southern Africa, waged between 1899 and 1902, which first alerted the British government to the need for establishing a permanent intelligence service. The so-called Boer War exposed to Britain’s military leaders, the Chiefs of Staff in London, how fragile the nation’s colonial holdings were. It took the British military much longer than expected, three years, and also the deployment of some 45,000 troops, to defeat a group of rebellious Dutch Boer farmers in the Cape Colony (now South Africa) who harried the British Army through guerrilla warfare. In fighting the insurgency there, it has to be noted that the British military developed some ominous strategies, not least the establishment of ‘concentration camps’, or detention camps, where suspected insurgents were ‘concentrated’. This type of warfare, in which the distinction between combatants and non-combatant civilians was blurred, was to have horrific echoes in the twentieth century. As far as intelligence was concerned, the kind of irregular warfare that Britain faced in the Boer War, like that experienced by other European powers in their own colonial ‘small wars’ – literally guerrilla in Spanish – revealed the paramount need for effective intelligence-gathering. In fact, it was during the Boer War that a British officer, Lt. Col. David Henderson, wrote an influential paper for the War Office in London, ‘Field Intelligence: its principles and practice’, which became the basis of a manual, ‘Regulations for intelligence duties in the field’, published by the War Office in 1904. This manual became the inspiration for the British Army’s intelligence corps, founded ten years later, on the outbreak of the First World War.⁶

Despite Britain’s long history of intelligence-gathering, a watershed occurred in the early twentieth century. Partly in response to fears of Britain’s colonial frailty, as revealed by the Boer War, but more specifically as a result of fears about the growing threat posed by the German empire, in October 1909 the British government took the momentous decision to establish a permanent, peacetime intelligence department. This decision was taken by the Committee of Imperial Defence – significantly, it was imperial defence that led to the setting up of Britain’s spook agencies. The department, known as the ‘Secret Service Bureau’, was divided into two branches. The ‘domestic’ branch, MO5(g), was responsible for security intelligence – counter-espionage, counter-sabotage and counter-subversion. During the First World War MO5(g) was renamed Military Intelligence 5, or ‘MI5’, and after the war it was again rechristened the Security Service – twin designations (the Security Service, MI5) that it keeps to the present day. Sir Vernon Kell, a retired officer from the South Staffordshire Regiment, served as Director-General of MI5 from 1909 to 1940, roughly one-third of its history to date, making him the longest-ever serving head of any British government department.⁷

Meanwhile, the ‘foreign’ branch of the Secret Service Bureau, first known as MI1C, was renamed Military Intelligence 6, or ‘MI6’, during the First World War. Thereafter it became known as MI6 or the Secret Intelligence Service (SIS) – again, twin designations that it retains to the present day. Its first head was Sir Mansfield Cumming, a Royal Navy officer who had taken early retirement due to ill-health. By all accounts he was a remarkable character. In the early stages of the First World War he lost a leg in a road traffic accident in France – as the story goes, he hacked his own leg off with a pocket penknife in order to drag himself to safety from the wreckage of his car. This accident caused him to use a wheelchair, and colleagues later recalled that he would terrorise the corridors of power in Whitehall, spinning at high speeds around corners.⁸

In taking the decision to establish a professional intelligence department in 1909, the British government actually came late to the ‘intelligence game’ when compared to other European powers, most of which had already set up such bodies by the turn of the twentieth century. France had established code-breaking ‘black chambers’ (cabinets noirs) in the middle of the nineteenth century, while tsarist Russia had an infamous intelligence service (the Okhrana), and Germany had a specialised intelligence service (Nachrichtendienst) operating at least since the time of the Franco-Prussian war in 1870. The reason for Britain’s late arrival into the world of espionage was due to strong opposition from some Victorian and Edwardian politicians, who decried ‘intelligence’ as an inherently un-English pursuit: gentlemen ‘did not read each other’s mail’, went the phrase, and ‘espionage’ was not even an English word, as some liked to point out. It was better to leave such sordid exploits to the Continental powers, where they belonged.⁹

The formation of the two services that would later become known as MI5 and SIS represented a fundamental break with all British intelligence-gathering efforts up to that point. For the first time, the government had professional, dedicated peacetime intelligence services at its disposal. Operational distinctions between MI5 and SIS, particularly jurisdictional disputes over what constituted ‘domestic’ and ‘foreign’ territory, proved a thorny subject that would only be resolved over subsequent decades. Nevertheless, the crucial point is that, unlike all British intelligence-gathering efforts up to that point, after 1909 the government was equipped with independent intelligence bureaucracies, furnished with card-catalogue index registries, which brought together information from all available sources. Whereas previously the British military and various government departments, such as the India Office, had gathered intelligence and conducted espionage for their own purposes, often on an ad hoc basis, the services established in 1909 had two specific combined purposes: to gather and assess intelligence. They were also inter-departmental, that is to say they were meant to ‘service’ all British government departments with the intelligence they needed. Although MI5 and SIS grew out of Britain’s military intelligence department (MO5), they were different from the intelligence departments of the armed forces, which were not inter-departmental. All three of the armed services, the army, Royal Navy and Royal Air Force, would go on to maintain their own intelligence departments, but it is MI5 and SIS (and later GC&CS) that are usually understood to be Britain’s intelligence services, or, more amorphously, ‘British intelligence’. The establishment of MI5 and SIS also witnessed for the first time a distinction between various grades of classified information (or ‘intelligence’), such as ‘secret’ and ‘top secret’. Thus, while British government departments before 1909 had gathered intelligence, and would continue to do so thereafter, the breakthrough for the government was that after 1909 it had for the first time its own intelligence services.

To this day, MI5 and SIS retain many of the practices established in their earliest days. The Chiefs of SIS retain the designation ‘C’, a title that was first used by Sir Mansfield Cumming, which is variously understood to stand either for ‘Cumming’ or for ‘Chief’. Other SIS rituals established in its earliest times which continue to the present include a green light outside C’s room (indicating that C is busy), special green ink that is reserved for him alone to use, and the ubiquitous and sometimes pointless use of codenames. SIS reports are still referred to as ‘CX reports’, apparently meaning ‘C Exclusively’. Similar continuities also exist in MI5. The terms ‘Put Away’ (‘P/A’) and ‘Look Up’ (‘L/U’), for example, can be seen on the front of countless declassified MI5 records, indicating when a file has been looked up and then put away in a secure cabinet – both of which were terms used by Kell soon after his ‘Bureau’ was established. The same is true of ‘Nothing Recorded Against’ (‘NRA’), which refers to one of the most important, but least glamorous, activities that MI5 officers have undertaken since Kell’s time: when an MI5 officer has looked up an individual in the service’s central archive, but has found nothing incriminating.

Eccentric rituals and designations apart, MI5 and SIS also retain much more important legacies from their early history. From the outset of their operations it was established that neither would have any executive powers. In contrast to law-enforcement agencies such as London’s Special Branch at Scotland Yard, or the US Federal Bureau of Investigation (FBI), neither MI5 nor SIS has ever had any powers of arrest. Readers may be disappointed to learn that SIS officers have never had ‘licences to kill’. Instead, MI5 and SIS have always relied on police authorities in Britain, particularly Special Branch, to carry out arrests for them. This was a calculated strategy on the part of the Chiefs of Staff and the British government. By decoupling intelligence-gathering from law enforcement, policing, and executive action more generally, they hoped to avoid the establishment of a ‘police state’, which they feared would be created by providing the secret services with powers of arrest. They also seem to have concluded that policing is a very different activity from intelligence work, which is not necessarily concerned with either arrests or law enforcement. Intelligence-gathering involves acquiring information in an anticipatory, prophylactic manner – fragments of information from here or there which may or may not become important one day. This distinction between intelligence and policing continues to the present day, and in fact is one of the reasons why the FBI at the start of the twenty-first century is considered to be ill-equipped to deal with the threat of terrorism, which requires anticipatory intelligence, not policing.¹⁰

Despite the notable continuities in MI5 and SIS’s history from 1909 onwards, it would be erroneous to suppose that in the years immediately after their foundation they were anything like the services they would later become. It is a myth that ever since the sixteenth century ‘British intelligence’, like the British empire itself, grew steadily in size and influence, spreading its tentacles across the world. This myth was mostly derived from Edwardian spy novelists like Erskine Childers, whose writings, such as Riddle of the Sands, depicted a powerful British intelligence service actively thwarting its enemies both at home and abroad. The reality could not have been further from such fictions. For years after their establishment, both MI5 and SIS remained desperately short of resources. The diary of MI5’s first and longest-serving head, Sir Vernon Kell, reveals how perilous the organisation’s existence was in its early days. In the years before the First World War, MI5 and SIS were both run on shoestring budgets. At the outbreak of war in 1914, MI5 had an entire staff of just fifteen – including the office’s caretaker. Staff numbers in SIS were similarly meagre. During his first week on the job in 1909, the first Chief of SIS, Sir Mansfield Cumming, noted rather miserably in his diary that he sat alone in his new office, without the telephone ringing, and with no one visiting him because the Bureau was too secret to be listed in a Whitehall telephone directory. This was very much a one-man-and-a-dog operation.¹¹

The First World War led to the massive expansion of the machinery of Britain’s secret state – just as it did in all other major belligerent European powers. In fact, it is fair to say that the war was the event that created the modern national security state. Every fighting nation built up unprecedented surveillance systems, and the strains of ‘total war’, in which all of a country’s resources were mobilised towards the war effort, necessitated an enormous increase in security and surveillance, both in Britain and across its empire. Total war required total surveillance. All of the warring governments were equipped with vastly increased new powers of detention and investigation, particularly through mail interception. MI5’s staff expanded dramatically after 1914, growing from a handful on the outbreak of war to reach 844 in 1918, of whom 133 were officers, as opposed to other ranks, while its central registry of people and organisations grew from 17,500 card indexes in 1914 to over 250,000 cards and 27,000 personal files in 1918. The Defence of the Realm Act (DORA), passed soon after the outbreak of hostilities in 1914, provided MI5 and other departments of the British secret state with enormously increased powers of surveillance. This is illustrated by the fact that at the start of the war the Post Office employed a single censor to intercept, open and analyse mail, but by its end the Censor’s Office had grown to employ over 2,000 officials, each of whom opened on average over 150 letters per day. It was also during the First World War that MI5 became more than just a ‘domestic’ intelligence service, as it is sometimes still mistakenly understood to be, and made a claim to be an imperial service, responsible for security intelligence in all British territories across the globe.¹²

The First World War is often regarded as a European war, a view that is reinforced by the famous war poetry of the Western Front, which vividly captures the horrific realities of trench warfare, with thousands of men being sent to their deaths in conditions akin to hell on earth. In reality, however, from the outset it was a worldwide war. Contrary to what we might expect, the first shots fired by British forces on land in the war did not take place in Europe, but were fired on 12 August 1914 at a German wireless station in Togoland, and soon after the outbreak of hostilities it became a deliberate policy of the Prusso-German General Staff to incite revolution and subversion (termed Revolutionspolitik) in the colonial empires and ‘weak points’ of its enemies. In September 1914 the German Chancellor, Theobold von Bethmann Hollweg, told his Foreign Ministry: ‘England appears determined to wage war until the bitter end … Thus one of our main tasks is gradually to wear England down through unrest in India and Egypt.’ It is revealing that while the British used the term ‘Great War’, from the start the German military spoke of a ‘World War’ (Weltkrieg).¹³

In 1914 the German General Staff established a new department, the Intelligence Bureau for the East (Nachrichtenstelle für den Orient), attached to the Foreign Ministry, which was led by an aristocratic Prussian archaeologist and explorer, Max von Oppenheim. The exploits of Oppenheim’s Bureau read much like the fantastic accounts of dastardly German plots to stir up unrest in India found in John Buchan’s classic wartime espionage novel Greenmantle (1916). Buchan describes a fiendish plan by the Central Powers to incite revolt in the Middle East and India, which it falls to his heroes, Major Richard Hannay and his friend Sandy Arbuthnot, a master of foreign tongues and exotic disguises, to thwart. In fact, Buchan’s story was not as absurd as the author purposefully made it appear. Buchan served as a war correspondent and briefly as a military intelligence officer at British headquarters in France, where he would have had access to intelligence records. His novel was fictional in degree, but not in essence.¹⁴

The reality was that before the war, Germany had been carefully cultivating links with Turkey and other Middle Eastern countries, which acted as the gateway to British India. Beginning in the 1890s, the German emperor, Kaiser Wilhelm II, had sponsored the construction of the Berlin–Baghdad railway, and during a trip to Damascus in 1898 he went so far as to declare himself the ‘protector’ of all Muslims – though it is unclear what reaction this received. Oppenheim’s new intelligence Bureau was responsible for inciting revolt among Germany’s enemies, and at various times during the war it sponsored French pacifists and Mexican nationalists, and most famously – or infamously, depending on one’s perspective – it helped a Russian émigré called Vladimir Ilyich Ulyanov, more popularly known by his pseudonym, Lenin, to return to Russia in April 1917 in a sealed bomb-proof train, with ample funds, shortly after which he successfully instigated a revolution against the temporary Russian government. There is no evidence that Lenin was a German agent per se, but he was certainly sponsored by Oppenheim’s Bureau – though presumably Lenin himself would have argued that it was he who was playing the German intelligence services, not the other way around. Nevertheless, in many ways the Bolshevik revolution was the greatest success of the wartime German intelligence services. Meanwhile, the main targets of Oppenheim’s Bureau in the British empire were Indian and Bengali nationals, Irish republicans and Arab jihadists.¹⁵

On 5 November 1914, soon after the outbreak of hostilities, the Ottoman empire entered the war on Germany’s side, and largely as a result of pressure from the German government, the Turkish Caliphate issued fatwas ordering all Muslims to wage a holy war (jihad) against Britain and its allies. British War Office records reveal the extent to which the Chiefs of Staff in London were concerned about subversion in the Indian army, one-third of whose soldiers were Muslims. It was also not lost on the Chiefs of Staff, nor on MI5, that approximately half of the world’s then 270 million Muslims lived under either British, Russian or French rule.¹⁶

At the beginning of the war, India was the only part of the British empire that MI5 was in direct contact with, communicating with the Director of Criminal Intelligence (DCI) in Delhi, Maj. John Wallinger. Previously, the main responsibility for dealing with Indian ‘seditionists’ or ‘revolutionaries’ (members of the Ghadr ‘revolt’ party) had fallen to the London Special Branch, but in the course of the war MI5 increasingly took a lead in dealing with Indian revolutionaries in Britain. After 1914 the German Foreign Ministry established an ‘Indian Committee’ in Berlin, which revolved around the exiled Indian academic and lawyer Virendaranath Chattopadhyaya, who had become a revolutionary while studying law at Middle Temple in London, and was a close confidant of the man who would later become the first leader of independent India, Jawaharlal Nehru. One of the agents being run by Chattopadhyaya in wartime Britain, Harish Chandra, was identified by MI5 through intercepted communications and interrogated by MI5 officers in October 1915. They persuaded him to act as a double agent, and he duly passed over considerable amounts of information on German plots in India. Reassuringly for MI5 and the Chiefs of Staff, the intelligence produced by Chandra revealed that the German Foreign Ministry was making increasingly unrealistic and far-fetched plans for subversion in India. The intensive interception of the mail of 138,000 Indian troops serving on the Western Front likewise convinced MI5 and the War Office that there was no widespread support for revolutionaries or for pan-Islamism among those soldiers – though one censor did report a worrying trend among them to write poetry, which he considered ‘an ominous sign of mental disquietude’. It was judged that the best strategy was to let the German Foreign Ministry continue wasting time, money and energy on fruitless plans for subversion in India.¹⁷

MI5’s main wartime expert on Indian affairs was Robert Nathan, who joined the organisation in November 1914, having spent twenty-six years in the Indian Civil Service and also serving as Vice Chancellor of Calcutta University before he was forced to resign due to ill-health. Within MI5, Nathan surrounded himself with a number of veterans of the Indian army, police and civil service. By 1917, MI5’s G-Branch, which was responsible for investigations, had a total of twenty-eight officers, eight of whom had previously served in India. This was an unusually large collection of Indian veterans for any British government department outside India itself. One of Nathan’s continual wartime concerns was possible political assassinations on British soil. In July 1909 an Indian Ghadr revolutionary had assassinated Sir William Curzon Wyllie, a former Indian Army officer and aide to the Secretary of State for India, on the steps of the India Office in London. Based in part on information provided by its double agent Chandra, MI5 feared that similar attempts might be made during the war. No such plot ever materialised, but MI5 continued to intercept and scrutinise the correspondence of known revolutionaries in London. In the spring of 1916 Nathan travelled to the USA, where his intelligence provided the US authorities with much of the evidence used at two major trials of the Ghadr movement, the first of which was held in Chicago in October 1916 and ended with the conviction of three militants. The second trial, held in San Francisco, came to a dramatic climax in April 1918 when one of the accused, Ram Singh, shot the Ghadr leader Ram Chandra Peshawari dead in the middle of the courtroom. The head of Special Branch in London, Basil Thomson, commented:

In the Western [United] States such incidents do not disturb the presence of mind of Assize Court officials: the deputy sheriff whipped an automatic from his pocket, and from his elevated place at the back of the court, aiming above and between intervening heads, shot the murderer dead.¹⁸

The Indian National Congress – the political body that would later become the main vehicle for anti-colonial nationalism in India – does not seem to have attracted any significant attention from MI5 or any other section of the British intelligence community during the First World War. This was partly because Congress had no significant wartime German connection, but also because before 1914 it was little more than a middle-class debating society that met only sporadically. There was nothing to suggest that it would emerge from the war as a mass movement that would become a focus for resistance to the British Raj. The main transformation of Congress’s fortunes would be due to Mohandas Karamchand ‘Mahatma’ Gandhi, an English-educated barrister of Inner Temple, who more than anyone else would set in process the downfall of the British empire in India a generation later. Nevertheless, in retrospect it is clear that in the pre-war and wartime years, MI5 and British intelligence authorities in India showed a remarkable lack of interest in Gandhi. When he returned in 1915 to India from South Africa, where he developed the technique of satyagraha, or passive resistance, which he later used against the Raj, the Department of Criminal Intelligence in Delhi described him as ‘neither an anarchist nor a revolutionary’, but just as a ‘troublesome agitator whose enthusiasm had led him frequently to overstep the limits of South African laws relating to Asiatics’.¹⁹

Reflecting the worldwide nature of the war, that same year, 1915, the British Chiefs of Staff instructed MI5 to establish a department to deal with German-sponsored subversion in the British empire. This new department, D-Branch in MI5, led by an officer called Frank Hall, expanded rapidly, so that by 1917 it had nineteen full-time officers. For cover, D-Branch used the name ‘Special Central Intelligence Bureau’ when communicating with colonial and Commonwealth governments. According to a post-war report compiled on D-Branch, its responsibilities included undertaking visa checks on individuals travelling in the empire and providing colonial governments with information on known and suspected German agents. By 1916 D-Branch was in touch with ‘the authorities responsible for counter-espionage in almost every one of the colonies’. However, during the war MI5 did not actually station officers in British colonies or other dependent territories overseas. Instead, it operated as a ‘clearing house’ for security intelligence on German espionage and subversion by maintaining direct personal contacts with a number of colonial police forces, known as its ‘links’, and consolidating all the information it received from those links across the empire into a single registry, which contained around 45,000 records in 1917. By the end of the war MI5 could justifiably boast that it presided over a unique, empire-wide index of security intelligence information. One of the clearest manifestations of its dramatically increased responsibilities was its famous ‘Black Lists’ of German agents, which it circulated to all colonial and Commonwealth governments. These grew from a single volume in 1914 to a weighty twenty-one volumes in 1918, which included 13,524 names.²⁰

The most famous example of German attempts to sponsor subversion in the British empire involved British counter-measures orchestrated by the now-legendary figure T.E. Lawrence, ‘Lawrence of Arabia’ – Oxford historian, archaeologist, cartographer, linguist, intelligence officer and expert guerrilla fighter. Lawrence and his colleagues, a group of self-proclaimed ‘intrusives’, established the so-called Arab Bureau in Cairo in 1914, which pioneered the use of guerrilla warfare against Turkish forces fighting on the side of Germany and the Central Powers in Arabia. In many ways the efforts of Lawrence and British forces in the Middle East during the First World War represent the first modern intelligence war: Lawrence’s forces combined intelligence gained from human agents with intelligence from signals radio intercepts, processes which are now known as human intelligence (HUMINT) and signals intelligence (SIGINT) respectively. The use of aerial reconnaissance, now known as image intelligence (IMINT), was also pioneered by the Royal Flying Corps in the clear skies over Arabia.²¹

The most important success of the Arab Bureau in its four-year wartime existence was to convince the War Office in London not to send an expeditionary force to Arabia. Lawrence and others in the Arab Bureau argued that a permanent British force landed in the Hejaz, the rocky province bordering the Red Sea, would inevitably be regarded by Arabs as an invading Christian, ‘crusader’ force – with disastrous consequences. Instead, the Arab Bureau insisted that the British should forge an alliance with the local Hashemite dynasty, who could take primary responsibility for fighting the Central Powers in Arabia, with the British providing assistance through intelligence and irregular warfare. This is precisely what occurred. British forces, led by Lawrence, collaborated with the Sharif of Mecca, Hussein bin Ali – who claimed to trace his descent back to Mohammed and Adam – and waged a series of spectacularly successful guerrilla attacks on Turkish and German forces in the Sinai Peninsula, conducting diversionary raids on railways and assaults on isolated garrisons in the Hejaz, through Aqaba to Amman and Damascus. In his famous self-dramatising account of his wartime exploits, Seven Pillars of Wisdom, Lawrence emphasised that intelligence-gathering was the key to successful irregular warfare: ‘The first line of guerrilla warfare,’ he wrote, ‘is accurate intelligence.’ Lawrence’s wartime mission was to divert Turkish forces from Palestine to protect the Hejaz railway. Assisted by Lawrence’s diversionary actions in the Hejaz, the leader of the Egyptian Expeditionary Force, General Edmund Allenby, won successive victories in Gaza and Beersheba, which ultimately led to the capture of the city of Jerusalem in December 1917. Allenby was joined by Lawrence in triumphantly entering the Holy City on foot – the first Christian soldiers to capture the city since the time of the Crusades.²²

As well as attempting to incite subversion against the British in India and the Middle East, the German military also sponsored unrest in one of Britain’s closest imperial possessions: Ireland. In fact, one of the most notorious cases that British intelligence was involved with in the entire First World War related to German attempts to forge an alliance with dissident Irish republicans. During the war, the signals department of the British Admiralty, codenamed ‘Room 40’, led by Reginald ‘Blinker’ Hall, grew in importance and size, and successfully intercepted and read many German communications. As with the rest of the British intelligence community, the war transformed the scale and nature of the British SIGINT, institutionalising code-breaking in ways that had not previously existed in Britain. One of the most notorious wartime German communications intercepted and circulated by Room 40 was the so-called ‘Zimmermann telegram’ of January 1917, in which the German Foreign Minister, Arthur Zimmermann, offered the Mexican government the chance to retake lost territories in the United States, including land in Texas, Arizona and New Mexico, if it declared war on the United States. Although the Zimmermann telegram is commonly known to be one of the causes of the US government’s entry into the war, the role that Room 40 played in the episode is still generally not properly appreciated in most histories of the First World War, despite having been discussed by intelligence historians for over thirty years: Room 40 intercepted the notorious telegram and passed it on to US authorities, who then publicly exposed it, while keeping secret the role of British code-breakers.²³

In the two years after outbreak of war in 1914, Room 40 code-breakers decrypted at least thirty-two German cables relating to Irish nationalists. The most important related to the ‘Easter Rising’ of April 1916, and Germany’s support of an exiled Irish nationalist, Sir Roger Casement, to help carry out the uprising in Dublin. Room 40’s decryption efforts gave the British government foreknowledge of the uprising, and provided exact knowledge of Germany’s arms supplies to Ireland. On information provided by Room 40, in April 1916 the Royal Navy intercepted the U-boat carrying Casement to Ireland before he could carry out his mission. Casement was ultimately executed by the British in Dublin in August 1916. During his incarceration he begged the British authorities to allow him to communicate with the leaders of the uprising, and warn them to abandon their plans. However, it seems doubtful that even if he had made such an intervention it would have prevented the council of the Irish Republican Brotherhood from proceeding. Furthermore, we now know that the controversial ‘Casement Diaries’ – lurid details of which, including graphic descriptions of Casement’s homosexual exploits, were deliberately released by Blinker Hall to blacken Casement’s name during his trial – were not forgeries concocted by the British, as many Irish nationalists at the time and since have maintained, but were in fact genuine. The attempts by the German government to incite an anti-British ‘fifth column’ in Ireland, the ‘back door to England’, was a strategy that would be repeated by Hitler a generation later, in the Second World War.²⁴

While the British secret state expanded rapidly during the First World War, the years after the war saw an equally quick deterioration of its resources. MI5 and SIS both had their budgets slashed, and MI5’s staff shrunk from 844 in 1918 to just twenty-five officers in 1925. It should be stressed that the dwindling resources of the British secret state after 1918 lay in sharp contrast to many states in post-war Europe, which turned to various forms of bloody authoritarian rule. In Soviet Russia, Fascist Italy and Nazi Germany, secret police forces expanded rapidly, and their leaders applied security practices forged during the First World War, such as mass registration and detention, to their populations even in times of peace. While the British government had interned at least 32,000 ‘enemy aliens’ on security grounds, largely on MI5’s advice, between 1914 and 1918, such drastic methods were taken only as wartime ‘emergency measures’ in Britain. The ‘totalitarian’ states of Nazi Germany and Soviet Russia, however, which were effectively at war all the time, and mobilised their populations along wartime lines, applied wartime security practices even while at peace. The head of Germany’s intelligence service in the First World War, Walter Nicolai, summarised this view when he wrote that for Germany to become a world power again it had to behave as though it was at ‘war in peace’, and would have to gather intelligence on all its enemies, at home and abroad. By contrast, the resources of the British secret state were cut so dramatically in the post-war years that by 1925 MI5 had a total of only twenty-four mail and telephone interceptions operating in the whole of Britain. Its staff consisted of just sixteen officers by 1929, and as MI5’s in-house history noted, its resources were so inadequate in the 1920s that its operations ‘reduced to a minimum’. Between 1919 and 1931, MI5 was relegated to investigating subversion in the British armed forces.²⁵

Despite the dwindling of its resources, throughout the 1920s and 1930s MI5 continued to pride itself on its imperial responsibilities established during the First World War. During the interwar years the overwhelming thrust of MI5’s duties was focused on the threat posed by the Soviet Union. With hindsight, we can see that MI5 and SIS both miscalculated, and were slow to react to, the growing threat posed by Nazi Germany after Hitler came to power in 1933. That said, MI5’s focus on the Soviet Union was hardly irrational. Soon after the Bolshevik revolution in 1917, the Soviet Union became just as much of an empire as the former Russian tsarist empire which it replaced – and posed a similar threat to the British empire. Immediately upon coming to power, Lenin pledged his support for a worldwide revolution against imperialism, which he famously described as the ‘highest form of capitalism’. The threat posed by the Soviet Union to the British empire was exposed by the head of the Intelligence Bureau in Delhi, Sir David Petrie, who went on to become the Director-General of MI5 from 1941 to 1946. In the late 1920s he wrote a classified official history, Communism in India, 1924–27, the circulation of which was limited to a small number of senior British officials in London and Delhi. Petrie warned that the Soviet Union posed a double threat for the British empire, especially in India: Soviet expansion was a strategic threat along the traditional lines of the ‘Great Game’ with Russia, but it also posed a subversive threat, with Moscow supporting anti-colonial movements inside the British empire. As it turned out, Petrie’s forecast was remarkably accurate: for over seven decades following the Bolshevik seizure of power in 1917, interrupted only by the Second World War, the Soviet Union became the main supporter of anti-colonial movements in the British empire and other European colonial empires. The reality was therefore that throughout the 1920s and 1930s the British government was engaged in a low-level ‘cold war’ with the Soviet Union, long before the real Cold War set in after 1945.²⁶

During the interwar years, Britain’s armed forces (the army, Royal Navy and RAF) all maintained intelligence departments that focused on the empire and Commonwealth. However, it was MI5 that was responsible for overall imperial security intelligence, maintaining direct contact with British intelligence authorities in India, the Delhi Intelligence Bureau (DIB), or Intelligence Bureau (IB) as it was also known, through a small London-based outfit known as Indian Political Intelligence (IPI). IPI was officially part of the India Office, which paid its budget, but from 1925 onwards it was housed in MI5’s headquarters at 25 Cromwell Road in London, opposite the Natural History Museum. The office space that MI5 rented to IPI (at rates that IPI often considered exorbitant) was by any standards pitiful: it consisted of three small, low-ceilinged, poorly-lit, dingy rooms in the attic of MI5’s headquarters. According to IPI’s head, Sir Philip Vickery (London Club: ‘East India’), these quarters were of such poor quality, with a ‘minimum amount of light and airspace’ and with half the main room ‘in darkness’, as to be almost ‘uninhabitable’. IPI was run on such a minuscule budget from the India Office that in 1926 it had only a handful of officers in London, equipped with three secretaries, one clerk and one typist, and apart from a few liaison officers in India, it had only two officers stationed elsewhere abroad, in Paris and Geneva. These few personnel formed the entire official security intelligence liaison channel between British intelligence and authorities in the subcontinent of India. Just how ill-equipped IPI was can be seen from a report in 1927: Vickery had to campaign hard for the purchase of ‘one extra hanging lamp’ for IPI’s attic office – for which the Department of Works ultimately seems to have refused to pay. When MI5 moved to its new headquarters, ‘Thames House’, Millbank, near Lambeth Bridge, IPI came with it, though its new accommodation was no better than before.²⁷

Despite the scanty resources at its disposal, sharing an office with MI5 allowed IPI to collaborate closely with other sections of British intelligence. Throughout the 1920s and 1930s, MI5 and IPI continued to monitor the activities of Indian revolutionaries in Britain. Their main subject matter increasingly became communist agents travelling between Britain and India on behalf of the Communist International (Comintern), an underground network that Moscow had established in March 1919 to act as a vehicle to export the ‘workers revolution’ from the Soviet Union to countries abroad. MI5 and IPI’s investigations were focused on detecting agents acting as secret couriers for the Comintern, passing information between the British Communist Party and communist cells in India. The main investigative mechanism they relied on was a Home Office Warrant (HOW), which allowed for the interception of telephone and postal communications. Unlike SIS, which operated abroad and collected intelligence from foreign countries illegally, MI5 and IPI’s area of operations, domestically in Britain and also in the empire, were constrained by law in ways that did not apply to SIS. This was the case even though at the time MI5 (and IPI) did not have any powers at its disposal, either in statute or in common law, which allowed it to intercept mail. Despite existing in a shadowy legal netherworld, MI5 records reveal that it worked hard to operate hard within a legalistic framework, even if not a legal one.

In order to impose a HOW, MI5 had to apply to the Home Office, with a written explanation of why a warrant was sought, and the Home Secretary then had to sign off on it. HOWs were extremely resource intensive – hence the small number that were operating in the 1920s. The actual interception of communications was carried out by a small, secretive section of the General Post Office (GPO), known as the ‘special censor section’, whose workers had all signed the Official Secrets Act. This section’s work was laborious and far from glamorous: its office was equipped with a row of kettles, kept almost continually boiling, which were used to steam open letters, after which their contents were photographed, resealed and sent on their way. Some of these intercepted communications, still found in MI5 records today, contain information about the private lives of MI5 targets and their broader social history that cannot be found in any other archive. Telephone calls were likewise intercepted (‘tapped’) by the GPO, which employed a small team of transcribers at the main telephone exchange in Paddington, London. The team included foreign-language speakers, especially ‘White’ (anti-Bolshevik) Russian émigrés, who translated telephone conversations in Russian and other Eastern European languages. Later, MI5 and the GPO developed an innovative device, based on a modified gramophone machine, which was used to record telephone conversations. This machine allowed record discs to be mechanically added and taken off the recording device, or ‘pooled’, thus eliminating the hitherto tedious task of GPO workers having to switch them manually.²⁸

The first agent identified by MI5 as acting as a Comintern courier was Percy Glading, a member of the British Communist Party who in 1925 travelled to India under the alias ‘R. Cochrane’. Glading’s covert trip was revealed to MI5 and IPI by intercepted communications through a HOW. Over the following years he also used his secretary, a pretty twenty-five-year-old blonde named Olga Gray, to deliver funds to communists in India. However, unknown to Glading or anyone else in the British Communist Party, Olga Gray was in reality an undercover MI5 agent, who had been recruited and planted into the British Communist Party in 1931 by MI5’s legendary agent runner, Maxwell Knight, one of MI5’s most successful counter-espionage officers in the twentieth century. After retiring from MI5 as a spymaster in 1946, Knight embarked on a highly successful broadcasting career, becoming known as ‘Uncle Max’, a colourful presenter of children’s radio nature programmes. According to a later report on Knight’s agent-running section, ‘M Section’, the six-year penetration of Gray into the British Communist Party had been so successful that she had achieved ‘the enviable position where an agent becomes a piece of furniture, so to speak: that is, when persons visiting an office do not consciously notice whether the agent is there or not’.²⁹

Olga Gray’s courier mission in 1935 to India for the British Communist Party provided MI5 and IPI with an extraordinary insight into how Comintern agents were run, and also revealed the identities of communist agents in India. However, her trip was an extremely delicate task for MI5, which had to go to remarkable lengths not to blow her cover. As Maxwell Knight later recalled, it was so badly organised by the British Communist Party that without MI5’s help it is unlikely that she would ever have got to India. Knight helped her devise a suitable cover story – that she was a prostitute – without making it appear that she had received help in concocting it. He also feared that if her passport and other paperwork for travel to India were approved too quickly, her superiors in the Communist Party might become suspicious. Her MI5 handlers therefore ensured that it was delayed sufficiently not to arouse any suspicion. After her trip, Gray revealed to MI5 the existence of a substantial Soviet espionage network operating in Britain. Its ringleader was none other than Percy Glading, and it was based at the Woolwich Arsenal in London, where Glading worked as a mechanic, and where he and his agents gained access to sensitive information on British armaments.

The strain of acting as a double agent began to take a toll on Olga Gray – she appears to have had at least one nervous breakdown – so in 1937 MI5 decided to wind up the Soviet network at the Woolwich Arsenal and have its agents arrested. Gray testified at Glading’s trial for espionage at the Old Bailey in February 1938, appearing anonymously behind a screen as ‘Miss X’. Her evidence helped to convict him of spying for Soviet intelligence, for which he was imprisoned for six years. The trial judge congratulated her for her ‘extraordinary courage’ and ‘great service to her country’. Soon afterwards, she left for Canada under a new name.³⁰

As well as providing intelligence on Soviet networks in India and Britain, Olga Gray’s position in the British Communist Party – unassuming but central – provided her, and thus MI5, with unique access to codes used by the Party to send radio messages to Comintern networks in Europe. Her information helped the Government Code and Cypher School (GC&CS), Britain’s first official peacetime SIGINT agency, established in 1921, to break radio traffic messages passing between the headquarters of the Comintern in Moscow and its numerous representatives abroad, in countries as far apart as China, Austria and the United States. GC&CS gave this radio traffic the codename ‘Mask’. The Mask traffic revealed to the British government that Moscow provided secret subsidies to the British Communist Party and also to its newspaper, the Daily Worker. In January 1935 Mask revealed the existence of a secret radio transmitter, based in Wimbledon, in south-west London, which was being operated by a member of the British Communist Party’s underground cell to send messages to Moscow. MI5 closely monitored the activities of those agents identified.³¹

MI5 and IPI identified other Comintern couriers, such as British Communist Party member George Allison, alias ‘Donald Campbell’, who, following a tip-off from MI5, was arrested in India in 1927 for travelling on a forged passport. However, the most important direct involvement of British intelligence in the empire at this time was with the so-called ‘Meerut conspiracy case’, a long-drawn-out trial that opened in India in 1929. Although their involvement was not publicised, both MI5 and IPI provided crucial evidence of the Comintern’s attempts to use communist agents in India to incite labour unrest there. In August 1929 the Deputy Director of MI5, Sir Eric Holt-Wilson, led a delegation of British officials to India to provide evidence at the trial and to testify to the authenticity of the intercepted documents – thus overcoming any objections the defence counsel might raise that the documents were unreliable ‘hearsay’ evidence, and should be inadmissible. The delegation, travelling First Class by ship and train, included five London Metropolitan Police Special Branch officers, as well as the head of the special censor section of the GPO, Frederick Booth, and the official in charge of the team in the GPO that actually photographed the documents, H. Burgess. They liaised closely with Sir David Petrie at the IB in Delhi, and judging from existing IPI records, it also appears that GC&CS provided intercepted communications passing between Moscow and a communist cell operating in India.³²

After providing evidence at the Meerut trial, Sir Eric Holt-Wilson embarked on an enormous worldwide tour, visiting and liaising with security officials from Hong Kong to New York. Holt-Wilson’s extensive trip was all the more remarkable given that it was made in an age before long-distance air travel, when the journey from Britain to India took weeks. More than any other MI5 officer in the first half of the twentieth century, Holt-Wilson – nicknamed ‘Holy Willy’ on account of his strong Anglican beliefs and because he was a rector’s son – was responsible for promoting the idea that MI5 was an imperial agency. In fact, he often referred to it as the ‘Imperial Security Service’. Holt-Wilson returned to India in 1933, at the conclusion of the Meerut trial, which led to the prosecution of a number of communist agents. Upon his return to London the next year he gave a closed lecture to the London Special Branch, at which he emphasised MI5’s imperial responsibilities:

Our Security Service is more than national; it is Imperial. We have official agencies cooperating with us, under the direct instructions of the Dominions and Colonial Offices and the supervision of local Governors, and their chiefs of police, for enforcing security laws in every British Community overseas.

These all act under our guidance for security duties. It is our duty to advise them, when necessary, on all security measures necessary for defence and civil purposes; and to exchange information regarding the movement within the Empire of individuals who are likely to be hostile to its interests from a security point of view.³³

Holt-Wilson went on another extensive overseas journey in 1938. The main purpose of this trip was to review local security and intelligence services in India and a number of other colonies and Dominions, and ensure that their security standards were adequate to meet the needs of the looming war with the Axis Powers. However, during the trip he himself displayed a remarkable disregard for basic security procedures – certainly far less care than he was attempting to instil in the colonial authorities he visited. In a series of soppy love letters that he sent by open, unsecured post back to his wife – a vicar’s daughter twenty years his junior – in England, Holt-Wilson described in detail the local intelligence officials he met, and also lamely attempted to glamorise for her benefit the nature of his ‘cloak and dagger’ work. If these letters, found in his personal papers now held in Cambridge, had been intercepted by the Axis Powers, they would have revealed a range of sensitive information on British imperial security and intelligence matters. The fact was that Holt-Wilson, a keen huntsman and one-time President of the Ski Club of Great Britain, was not one for modesty – which is surprising for someone whose career necessitated working in the shadows. In his own words he was ‘a champion shot’, and in the official description he penned for himself in Who’s Who, he stated that he was the Director-General of the ‘Imperial Security Intelligence Service’, and also accurately but pompously noted that he was ‘author of all pre-war official papers and manuals on Security Intelligence Police Duties in Peace and War’. Not very subtly for one of Britain’s senior intelligence officials, Holt-Wilson also listed his home address in his Who’s Who entry. ³⁴

In March 1938 Holt-Wilson arrived in India, where he met the new head of the IB in Delhi, Sir John Ewart, whom he referred to as the ‘K [‘Kell’] of India’. He next travelled to Singapore and Hong Kong, where as he reported to his wife, he was spotted by local press reporters as being involved with ‘hush-hush’ work. In Singapore he liaised with a local MI5 officer stationed there, Col. F. Hayley Bell, in Holt-Wilson’s unflattering opinion a ‘deaf madman’, whose deafness made hushed conversations difficult. He also met Hayley Bell’s daughter, Mary Hayley Bell (later Lady Mills), who in 1942 would write a popular wartime play, Men in Shadow, about resistance groups in France, which would attract the attention of MI5 for revealing sensitive details of escape routes from occupied France. MI5 only allowed the play to be performed after the passages in question were removed. At a dinner held in his honour during Holt-Wilson’s visit to Hong Kong in April 1938, which was officially described as an ‘inspection of the colony’s defences’ so as not to attract too much press attention, the Governor proposed a toast to ‘good old Thames House’ (MI5’s headquarters), which was lost on all the guests except for himself and Holt-Wilson.³⁵

Ireland was a particularly important recruiting ground for colonial police officers, many of whom would deal with intelligence matters across the empire. After the Irish Free State was granted a form of Dominion status in 1921, a stream of former officers of the disbanded Royal Irish Constabulary (RIC) moved into the Indian and other colonial police forces, particularly in Palestine, where they gained a reputation for stern discipline and ‘backbone’. Ireland was also the theatre that provided a model for policing and counter-insurgency operations that persisted in British military thinking for several decades. In 1934 Major General Sir Charles Gwynn published an influential book, Imperial Policing, on low-intensity conflicts or ‘small wars’. Drawing on lessons from Ireland, and the tactics the British used to crush the Indian Mutiny in the 1850s and other Indian revolts at Dinshawai (1906) and Amritsar (1919), Gwynn recommended that to be effective, colonial policing required the use of minimum necessary force, with the aim of restoring civilian government as soon as possible, and tactics such as troops moving in sweeping column formations against enemies. While Gwynn’s recommendations were undoubtedly applicable to Palestine in the 1930s, they left their mark for much longer than they should have on British military authorities, who continued to apply these tactics to anti-colonial insurgencies in the post-war years, when they were largely irrelevant because Britain’s enemies in those conflicts did not fight in open, regular and identifiable formations. Thanks in large part to Gwynn, there was a direct continuum between the way the British military crushed colonial revolts in India in the 1860s, and how it tackled post-war insurgencies in places like Palestine, Malaya, Kenya and Cyprus.³⁶