UK MKULTRA LinksPreview
Open in TheBrain

Paul Ekman

Abstract

This essay examines the detailed process of isolating facial data from the context of its emergence through the early work of psychologist Paul Ekman in the 1960s. It explores how Ekman's data practices have been developed, criticized, and compromised by situating them within the political and intellectual landscape of his early career. This essay follows Ekman's journey from the Langley Porter Neuropsychiatric Institute to New Guinea, highlighting his brief but notable collaborations with psychologist Charles E. Osgood and NIH researchers D. Carleton Gajdusek and E. Richard Sorenson. It argues that the different meanings assigned to the human face resulted in how each group developed their studies – examining facial expressions either in interaction, where they shape reciprocal actions in interpersonal communication, or in isolation, where faces surface from the individual's unconscious interior.

In his review of psychologist Paul Ekman's The Face of Man (1980), psychologist Silvan Tomkins praised the book as “a milestone in our understanding of human beings” (Tomkins, 1982, p. 854). The book was a compilation of Ekman's achievements in the universality of facial expressions and emotions over the past two decades. More than half of its pages were devoted to the photographs he took in the New Guinea village, primarily close-up shots of the South Fore faces. According to Tomkins, the facial expressions in each photograph served as “a testament to the universality of emotions in the family of man” (ibid). Tomkins's description of the photographs as a “testament” was further explained in Ekman's introduction to the book. If the readers found the Fore facial expressions familiar and recognizable “as the next door neighbor,” it was because facial expressions of emotions were the same for all humans regardless of culture, race, and sex (Ekman, 1980, p. 1). Ekman further scrutinized these photographs in his laboratory and observed that the South Fore subjects used the same facial muscles as those from other cultures when experiencing or posing for the same emotions (p. 10). For Ekman and Tomkins, these visual recordings were concrete evidence supporting the theory on universal facial expressions of emotions and served as a fundamental resource for a deeper exploration of the meaning of facial movements.

Visual media, such as photographs and films, have been used by a group of human and behavioral scientists to study human nonverbal behavior since the early twentieth century. Anthropologists, who saw bodily interactions as reflections of cultural patterns, used ethnographic films to document their subjects (Jacknis, 1988). Psychiatrists captured the interaction between psychiatric patients and their families on film to understand how disrupted communication could lead to the onset of mental illness (Geoghegan, 2017; Watter, 2017). Still photographs of actors performing facial expressions were used by experimental psychologists to explore the messages perceived by their observers (Watson, 2004; Bollmer, 2023). Visual media served as scientific instruments to capture and project transient movements into fixed imagery. Body movements and their interactions transformed into objects of scientific inquiry.

Ekman distinguished himself from his predecessors by taking a unique approach to these visual recordings, utilizing them as primary material for further deconstruction. The whole body was isolated from its situational context, which was then dissected into its compartments, such as hands, feet, and face. In the end, facial expressions were scrutinized by scoring their fundamental muscle movements (Ekman & Friesen, 1978). These new ways of seeing the face also brought changes to the expertise for studying human nonverbal behavior, transitioning from a subjective observation of facial expressions within the broader contexts to a mechanical transcription of their musculature units. The true meaning of the face was to be understood from the subtle cues that were imperceptible to the untrained and unaided human eye. With his techniques of abstracting, capturing, and fragmentizing bodily movement into analyzable forms, Ekman strived to bring his version of “scientific objectivity” to his scientific exploration of the essence of “being human” (Daston & Galison, 1992).

Ekman's theories and practices, however, have faced criticism for over half a century. Ekman has always been at odds with the “cultural relativists” who prioritized the cultural aspect of nonverbal behavior and were unconvinced by Ekman's assertion of pan-cultural facial expressions of emotions. This dichotomy between the universalist and the social constructivist perspectives has been the dominant narrative in the histories of emotion sciences. Within this narrative, the main opponents of the universalist and essentialist approach in human sciences were those who perceived facial expressions and emotions as social constructs. It provided a lens to understand the reason why cultural anthropologists with ethnographic experience in distant cultures have criticized Ekman for neglecting the context that shapes emotions and facial expressions, leading to skepticism about the credibility of his cross-cultural studies. In their view, Ekman's claim that he had control over the cultural conditions in his carefully orchestrated experiments was unrealistic, given the short duration of his visits (Plamper, 2015; Reddy, 2001). Other historians complicate this narrative, the polarized spectrum between the universalist-social constructivist divide, to conclude that the main distinction between the two camps was the question of whether human cognition shapes human emotion and facial expressions of emotions (Leys, 2010, 2017). These histories are significant for gaining insights into the overall landscape of emotion sciences and a thorough genealogy of Ekman and his collaborator's research.

This essay aims to contribute to this existing literature in the history of science by focusing on the unique practices of making and utilizing facial data by Paul Ekman, which was at the root of this conflict. It will pay closer attention to the “data practice,” which encompasses both the practice of making data, from choosing the object of interest to using statistical methods for analysis, and the implications of this data in practice.1 Historian Iris Clever used the concept to discuss various methods of abstracting humans into data that facilitated anti-racist campaigns and gave a measurable reality to biological race in the early twentieth century (Clever, 2023). Recording, storing, and analyzing human data from individual to nation and addressing Cold War concerns was a common pattern for behavioral scientists. Even though Ekman adhered to this pattern, he was indifferent to understanding the human face in interactions that structure interpersonal communication, a subject that fascinated those who came before him. Instead, he sought or brought the human face into isolation with new techniques and expertise, which gave facial expressions of emotions a measurable and isolatable presence. Through the isolated face, scientists began to understand facial expressions as emerging from the unconscious and universal interior of human individuals and sought clues to resolve the problems afflicting their societies (Herman, 1995, p. 173).

To understand how Ekman's data practices have been developed, criticized, and compromised, this essay positions his facial sciences within the political and intellectual landscape of the 1960s when Ekman was in his early career stage. I follow his collaborations with psychologists, psychiatrists, and anthropologists, from psychologist Charles E. Osgood to National Institutes of Health (NIH) researchers D. Carleton Gajdusek and E. Richard Sorenson. Osgood is renowned for his contribution to the birth of psycholinguistics and Cold War international politics (Cohen-Cole, 2014b; Lemov, 2013). His research interest in the language of facial expressions, which led to the planning of a cross-cultural study with Ekman, has not received enough historical recognition. Due to unexpected challenges in coordinating the project with Osgood, Ekman changed his plans and continued his cross-cultural study in New Guinea with Gajdusek and Sorenson, who saw great potential in cinematic films for the study of man (Link, 2016). The universalistic worldview and data practices of Paul Ekman's sciences were shaped by his interactions with these actors.

This essay will begin with Paul Ekman and Charles Osgood's early encounters in the early 1960s. As a visiting scholar at the Center for Advanced Study in the Behavioral Sciences (CASBS) in the late 1950s, Osgood developed a keen interest in international politics and began actively discussing the subject. This section will elaborate on how Ekman integrated Osgood's ideas into his own views on the relationship between science and politics. The second section examines the development of Ekman's theories and practices at the Langley Porter Neuropsychiatric Institute. It explores the details of his methods in producing data from film recordings by transforming transient body movements into a series of static positions. The third section traces Osgood and Ekman's collaboration and eventual dissolution in 1966. Central to their discussions were the contrasting ways of seeing the human face, either through a linguistic or a clinical lens. For Osgood, facial expressions represented a language in themselves, while for Ekman, the face was a promising source of data for understanding the human interior. The design of each study was contingent upon the meanings attributed to the human face, either the face in interaction or the face in isolation. The final two sections follow Ekman's new project with the NIH researchers and their journey to New Guinea in 1967 and 1968. In this changed environment, Ekman maintained his belief that the human face was a rich and reliable data source, which should be isolated from the context of its emergence. His data practices on the “Stone Age” culture, however, were at odds with the researchers who had extensive ethnographic knowledge and the Indigenous human subjects who had never experienced isolation or separation from their group.

1 CHARLES OSGOOD AND THE “ALTERNATIVE TO WAR”

Paul Ekman, a clinical psychology graduate student at Adelphi College in New York, went to the Bay Area in September 1957 to work as a clinical intern at the Langley Porter Neuropsychiatric Institute. Langley Porter offered an ideal setting for Ekman to develop his study of nonverbal behavior, given that it was where psychiatrist Jurgen Ruesch, known for his cybernetics-inspired theory of nonverbal communication, was the Director of Research. Although Ekman's own communication with Ruesch during the year-long internship was rather limited, his doctoral dissertation in 1958 closely reflected Ruesch's theory and provided empirical evidence of the reinforcing effects of nonverbal stimuli – such as a head nod, smile, and a slight movement forward – in comparison with verbal stimuli (Ekman, 1958). After graduation and 2 years of military service, Ekman returned to Langley Porter as a Postdoctoral fellow. For some years, his study of the semiotics of facial behavior closely reflected Ruesch's theory of the semiotics of human relations (Ekman, 1987; Schiller, 2021).

Ekman's activities at Langley Porter took an unexpected turn after attending a special lecture at Langley Porter delivered by psychologist Charles Osgood, a professor at the University of Illinois Urbana Champaign, in October 1961. Osgood was an early contributor to the new field of psycholinguistics and a vocal speaker in peace studies. His speech brought attention to the ways psychologists could participate in advancing the strategies of peace, which motivated Ekman and other attendees to come together to form their own peace studies group in the Bay Area. Shortly after, more behavioral scientists who were interested in promoting the “maintenance of peace” joined the group, which became the Committee for the Application of the Behavioral Sciences to the Strategies of Peace (ABSSOP).2 Ekman chaired the committee from 1961 to 1963, dedicated to addressing the critical social and political challenges of their society. Within 3 months, what began as an informal gathering of 25 members quickly grew to include 450 names on the mailing list. Charles Osgood, who had caused the “stir” with an impactful speech, also became its member.3

It wasn't until his residency at the CASBS in 1958 that Osgood began to discuss topics of international politics and nuclear tension. He co-led a seminar series entitled “The Implication of Social Science for Policy in a Nuclear Age” and released publications examining the concerns of escalating armament and national security (Osgood, 1959). In his work, Osgood pursued an “alternative to war,” a nonviolent approach to reducing international conflict. The possibility of a nonviolent approach was supported by his knowledge and expertise as a psychologist who believed that “Russian perceptions and reactions to the present situation are much like our own” (p. 315). Osgood believed that scientists such as himself could leverage the universal principle that governs the communicative minds of men to alleviate international tension, regardless of cultural or ideological differences. The CASBS marked the beginning of his journey as a Cold War behavioral scientist, committed to the scientific study of the biological, psychological, and social man while engaging with the pressing concerns of his community.4

Osgood's devotion to peace studies continued after returning to the University of Illinois. While serving as president of the American Psychological Association from 1962 to 1963, he worked as a member of a six-man working group in Project Michelson, an initiative directed by the Behavioral Sciences Group at the US Naval Ordnance Test Station.5 The project focused on providing the grounds for deterrence theory, which explored how the threat or limit of force by one party could deter the hostile actions of others. In the group, Osgood worked closely with Stanford political scientist Robert C. North, who was also a member of the ABSSOP. Incorporating North's expertise in international politics with his own work on the symbolic processes of Skinnerian behaviorism, which had been the main concern of his studies in human communication, they proposed a stimulus-response model for nations. The model was introduced in their co-authored report, “From Individual to Nation: An Attempt to Make Explicit the Usually Implicit Process of Personifying International Relations” (1962), which they circulated among their colleagues and relevant institutions. North later described it as a “precise formulation of the nature of any perceiving, interpreting and reacting entity – from the individual to the national level” (North et al., 1963, p. 4). This formulation is summarized in the following diagram (see Figure 1).

Figure 1

Open in figure viewerPowerPoint

General model of interpretation and evaluation by Osgood and North (Osgood & North, 1962; in North et al., 1963, p. 4, redrawn by the author of this essay).

The premise of the stimulus-response model of nations was that all human interactions, regardless of scale, could be understood through the lens of information processing, from decoding information to encoding decisions. Such vision aligned with the belief shared by many scientists that individuals, rather than groups or institutions, were the “proper units of analysis” in behavioral science research (Robin, 2001, pp. 6 and 25). Osgood pushed this notion forward to argue that the “principles of semantics discoverable at the individual level can be transferred to the national level,” as the so-called “behaviors of nations” were determined by the individuals who were decision-makers for the nation (Osgood, 1971, p. 44; Osgood, 1980, pp. 374–5). The recognition that a universal principle governed the interactions of various scales inspired his colleagues to contribute to developing strategies for peace utilizing their own expertise, with Paul Ekman among them.

As its chairman, Ekman was a part of various projects that took place within the ABSSOP committee. He worked as a research associate at Stanford Studies in International Conflict and Integration between 1963 and 1964 and collaborated with its director, Robert North, on the “Simulation of Interpersonal Conflict” project (North, 1962). The project was about moving beyond the conventional game theory and developing a game that simulates real-life interpersonal conflicts. Players who stood face-to-face took turns to roll their balls into holes with varying point values, providing them with more than two options for their moves. Because the game resembled a pinball machine and involved strategic thinking, it earned the moniker “the thinking man's pinball” (see Figure 2) (Billboard October 31, 1964; IEEE Spectrum, 1964, pp. 172 and 174).

Figure 2

Open in figure viewerPowerPoint

The design of the thinking man's pinball (Ekman & North, 1963, p. 9).

Ekman and North added another layer of “thinking” to the game, hoping to explore different theories on human conflict. They planned to recruit individuals “who explicitly disagree about values and goals” (Ekman & North, 1963, p. 16). Two people from different groups, each representing opposing ends of the political spectrum, would be paired together – one from the Young Americans for Freedom (a conservative youth activism organization) and the other from the Vietnam Day Committee (a coalition of left-wing political groups, student organizations, labor unions, and pacifist religions opposing the Vietnam War). During the pinball game, the players were to talk about a controversial subject and convince one another to change their opinions. Ekman believed that the conversation would evolve into an argument, which he would record, crop, and edit to use as stimulus materials in future studies (ibid.). The design of the game reflected the world divided into two, both internally within the nation and externally on a global scale. Its title, “Simulation of Conflict,” also suggested that it was simulating the era-defining conflict, the Cold War. Ekman and North's belief that their discoveries could contribute to a better understanding of human interactions ranging from individuals to nations closely mirrored the Osgood-North model of stimulus-response of nations.

2 COLLECTING FACES IN THE CLINIC AND BEYOND

While working part-time in the Stanford Studies Program, Ekman's regular position was at the Langley Porter Neuropsychiatric Institute as a postdoctoral research fellow. Ruesch's work in nonverbal communication had been a major influence on Ekman to expand his interest in the meaning of nonverbal behavior. Nevertheless, Ekman saw the work constrained by its lack of “empirical evidence” (Ekman, 1958, pp. 8–9). Ruesch's theory was developed from his careful observations and photographic evidence instead of the results from well-structured experiments. This problem was not exclusive to the work of Ruesch; it was a challenge faced by most studies in psychology that explored the subject of “knowing other people,” as they often lacked robust theoretical frameworks, experimental techniques, and empirical analyses (Bruner & Tagiuri, 1954). Ekman responded to these concerns by reverting back to the “traditional approach” that Ruesch had tried to move away from, which involved breaking down the interactions into smaller units of body movements (Ruesch & Kees, 1956, p. 13). He began by developing and incorporating empirical techniques to further develop the film-based inquiries of his predecessors.

Ekman's systematic exploration into nonverbal behavior, a topic that was not widely accepted in mainstream psychiatry or psychology at the time, was facilitated by Langley Porter's strong support for communication studies. Ekman's first project at the institute consisted of recording and analyzing face-to-face interviews between a graduate student and a senior psychologist. He soon joined the Family Study Project of the Rheumatoid Disease Group, where clinical interviews were captured on films (Ekman, 1963). The institute was already equipped with advanced technologies for analyzing film recordings by the 1960s. Behavioral sciences laboratories were connected to videotape laboratories, a computer center, and neurosurgical operating rooms through data transmission cables. There was also a research space dedicated to studies on human interaction with audio-recording facilities, time-lapse photography, closed-circuit television, and apparatus for analyzing human speech (Ruesch, 1978, p. 44). Ekman's research interest was shaped by the intellectual, institutional, and instrumental conditions of Langley Porter. The institute would soon recognize Ekman as having brought “new impetus” to Jurgen Ruesch's studies in nonverbal communication (p. 72).

Methodologically, Ekman's research was in conversation with a strain of interaction sciences that began in the mid-1950s, wherein scientists began to study talk therapies, small group conferences, and clinical interviews on films. With their machines that could record, playback, duplicate, and crop an interactive scene, scientists in search of patterns in behaviors were able to conduct a fine-grained analysis of their subjects of observation (Geoghegan, 2017; Lempert, 2019; Watter, 2017). How Ekman defined still and moving pictures is notable, as he contended that the motion picture camera only “takes more samples of positions per second.” (Ekman, 1957, p. 143). He believed that nonverbal behavior was essentially a display of movements occurring between one position and another. Still and moving cameras were instruments that captured and translated the movements of the face and body into fragments of body positions. This meant that the investigators required the skills to specify and isolate a critical position for displaying information from its mass. These skills would also become a subject of scientific analysis and technological support in Ekman's future studies.

Langley Porter provided another advantage to Ekman's research in terms of the information that was being conveyed through nonverbal cues: affective experiences of the hospitalized patients. In 1963, Ekman was invited by his colleagues to join their 5-min interview sessions with hospitalized depressive patients. Ekman translated this invitation into a valuable opportunity to witness the patients' emotional turmoils and “acquire daily photographic measures and a wealth of collaborative data” and launched a new project focused on the nonverbal behavior of psychiatric patients (Ekman, 1964). Working together with coinvestigator Charles Cress, Ekman filmed their interviews with patients at different stages of their hospitalization - upon admission, during hospitalization, and before discharge. These moments of hospitalization were, according to Ekman, “changes crucial to the life of the sender [patients]” and were “naturally occurring emotional experience[s]” that would stimulate the patients. The hospital environment itself became a part of Ekman's experimental settings wherein he could delve into the subject of nonverbal behavior of emotions (Ekman, 1965a, p. 7).

Ekman's data collection included two types of patients: depressive patients and schizophrenic patients. Depressive patients were chosen because of their affective disturbances and the rapid mood swings during the adjusting period. Schizophrenic patients, in contrast, were expected to show different results as their issues were more cognitive rather than affective (p. 19). In this sense, Ekman believed that the recordings made from these interviews captured the quintessential element of emotional expressions as they were made by emotionally disturbed subjects in an emotionally stimulating environment.

In an effort to convert films into empirical evidence, Ekman implemented strict regulations during their production. To ensure consistency of the recordings, each film featured a patient with a complete view of their face and body, seated against a standardized, grid-like background and dressed in identical clothing. Factors including camera angle, focus, exposure, lighting, room arrangement, furniture placement, and the distance between the patient and the interviewers were also meticulously controlled (see Figure 3). Patients were informed that the interview would be audio recorded on tape for research purposes and had given their consent to be photographed upon admission. They were not aware, however, that the entire interview process would be recorded with a 16 mm optical sound motion picture camera that was concealed and installed in another room. Each patient was filmed three times, 8 min each. To record an 8-min interview with 24 frames per second required an 864-foot film. Over time, the collection grew to include film recordings of 40 female patients, some diagnosed with depression, some with schizophrenia, and others with both. These original films served as prototypes that could be duplicated and then edited, allowing for a more targeted focus on specific body areas according to the purposes of their research. This reproduction and expansion of films ultimately transformed Ekman's collection into what he would describe as a film “library” (p. 100).

Figure 3

Open in figure viewerPowerPoint

Films of Langley Porter Psychiatric Patients 1963–1966. (PE Papers, Box 23, Folder: “Nonverbal Behavior and Psychotherapy Research” (1968) “13” × “16”).

Along with controlling the interview setting that would be captured on films, Ekman's data practices involved isolating the proper unit of analysis through these recordings. In his grant proposals and publications, Ekman made a direct quote from cybernetics-inspired anthropologist Gregory Bateson that “nonverbal behavior is largely a relationship” (p. 7). His emphasis on the interactive nature of nonverbal behaviors resonated with the growing awareness among psychiatrists and psychologists about the importance of therapist–patient communication and relationships, as well as the reflexive nature of their scientific endeavors (e.g., Capshew, 1999; Morawski, 2015; Shlien, 1968). Contrary to his understanding that nonverbal behaviors are social, interactive, and responsive, the focus of Ekman's camera moved away from exploring the patterns of interactions to solely capturing the physical movements of the patient's bodies, placing the interviewers out of the frame. As historian Lisa Cartwright argues, film technologies in early twentieth-century physiology transformed the human body into a dynamic entity that should be captured, isolated, and studied in motion, which she summarized as the “physiological gaze.” The knowledge practices of physiologists legitimized the usage of motion picture cameras in observing and scrutinizing the subtle changes taking place in human bodies, which was also the case for Ekman (Cartwright, 1995, p. xiii). By incorporating Bateson's anthropological observations into his “physiological gaze” toward psychiatric interviews, Ekman became convinced that one could disassemble interactive nonverbal behavior into its basic units. In the end, Ekman arrived at the conviction that a rigorous observation of this interactive and dynamic body could yield valuable insights into the internal state of humans.

Ekman had made another audacious move in his patient films – separating the head and different parts from the entire body. His question was whether the patient's head and body delivered the same messages regarding their emotional state. In his experiments, judges were given photographs that blocked out either the head or the body and were instructed to rate them on three nine-point scales: from pleasant to unpleasant, from tension to sleep, and from attention to rejection. Face-to-face interactions were broken down into communicative individuals and then into the movements of different body parts. From these studies, Ekman concluded that the head and body had different sending capacities and carried different types of information (Ekman, 1965c; Ekman & Friesen, 1969). It was through the assemblage of the psychiatric interview techniques, interactive body, and filming instruments that the surface of the face and body were reshaped into valuable sources of information. In light of his finding that the face was the best sender in terms of its visibility and the number of patterns shaped by its musculature, Ekman's subsequent research revolved around facial expressions exclusively.

Having established his research methodologies, Ekman explored the possibility of expanding his subjects of study beyond the confines of the ward. Proposing and leading the aforementioned Simulation of Conflict project at Stanford was the first step that Ekman took to implement his techniques to observe the interaction between non-patient players. In time, the human subject of Ekman's research came to include a broader spectrum of the “human.” From the belief that he could successfully “stabilize the subjects” by keeping all conditions under control and take a leap to interpret the results as general human experience, Ekman ventured into studying the nonverbal behaviors of unfamiliar and remote cultures (Martin, 2022; p. 138; see also Danziger, 1990).

3 “LANGUAGE” OR “INFORMATION”

From the early 1960s, Charles Osgood had a grand vision of expanding his studies on human language and concepts to create the “Atlas of Affective Meanings.” The original plan for the Atlas was to explore the fundamental affective meaning of different languages in various countries, such as the United States, Hong Kong, Finland, Japan, Mexico, the Netherlands, and Turkey.6 By 1964, Osgood came to discuss the possibility of including facial language in the Atlas. He made a proposal to Paul Ekman about their collaboration in a comparative study of facial expressions in a number of countries in their “network.”7 The Atlas would undoubtedly benefit from the inclusion of facial expressions as a form of communication.

A new funding opportunity reignited the conversation for Ekman about conducting a cross-cultural study of facial expression with Osgood. In November 1965, Ekman visited Lee Huff, the head of Behavioral Sciences at the Advanced Research Projects Agency (ARPA), to extend the funding for his Simulation of Conflict project. Huff suggested that Ekman write a grant proposal for a cross-cultural study on nonverbal behavior.8 This suggestion came as ARPA reduced its support for game or bargaining research and provided more opportunities for cross-cultural studies. The condition for this funding was that the project should be basic research, with no military application and no subcontracts. This shift in patronage was a direct result of the termination of Project Camelot, an initiative by the Department of Defense to study signs of revolutions and counterinsurgency tactics in various countries that needed special attention. After the project was discovered and publicized by South American scholars and media, the Chilean senate condemned the initiative as a form of imperialistic intervention. It provoked an international outcry that led to the project's cancellation in July 1965 (Horowitz, 1967). The research funds for Project Camelot were redirected to “noncontroversial” overseas projects supported by the Army, Navy, Air Force, and ARPA (Ekman, 1987, p. 15; Ekman, 2016). Given that Ekman's nonverbal behavior research did not exhibit any association with counterinsurgency or foreign intelligence projects, it emerged as a strong candidate for the new funding.

When Ekman stumbled upon this unexpected opportunity, Osgood was in Hawaii from 1964 to 1965, dedicating his sabbatical year to writing a series of papers on the meaning of facial expressions. Osgood and his coauthors were going to discuss a number of different experiments, including the one from his introductory psychology course in the 1940s. In the experiment, Osgood created a communication situation between an actor and the judges. The actor, who was randomly selected, chose 1 out of 40 “emotional states” written on a paper slip. The psychological process here was that the actor decodes the meaning of the emotional state on the slip and generates a facial “posture” for 10 s. The remaining students became judges who would observe the actor, decode the meaning of the actor's posture, and choose the best match from the same forty emotional states written on the blackboard (Osgood, 1955, 1966). In another experiment, Osgood and his group used photographs of actors posing for specific emotional states and had the subjects rate them according to the semantic differential technique, a method to measure the underlying meanings of each facial expression (Hastorf et al., 1966). As a psycholinguist who was interested in the dynamics of human perception and cognition in shaping the meaning of language, Osgood understood facial expressions as a linguistic phenomenon delivered through a nonverbal channel. Positioning the actors and judges to sit face-to-face or applying the technique used for language analysis to facial expressions indicates that the experiments were an evaluation of emotional messages sent and received through the face. It was an exploration of the faces in interactions.

With the opportunity to conduct cross-cultural research on facial expressions, Ekman immediately reached out to Osgood. The collaboration between Osgood and Ekman quickly gained momentum. Osgood's research group had been preparing for a comprehensive study on the semantics of facial expressions across different cultures. One option for Ekman was to join Osgood's team in Yucatan, Mexico, where they were actively collecting language data from the Mayan Indians.9 Following numerous phone calls and exchanges of letters, Ekman increasingly became dependent on Osgood to draft his ARPA proposal (Ekman, 1966). Working with Osgood, utilizing his semantic differential technique, and leveraging his extensive international network made the proposal both promising and feasible. Ekman also saw great potential in this collaboration as he was in search of an opportunity outside clinical psychology (Ekman, 2016, p. 86). The cross-cultural project became a critical juncture that would help him transition seamlessly to studying broader human experience from what Ekman referred to as “low” cultures and, ultimately, the “normals.”10

In the proposal submitted to ARPA, Ekman emphasized the advantages of analyzing faces and bodies in comparison with verbal behavior. According to Ekman, nonverbal behavior is difficult to censor, control, and mask, which makes it an important source of information. Most individuals do not understand the process of making judgments through this communication channel and do not pay much attention to it. Notably, individuals who excelled in these abilities included psychotherapists, intuitive or practiced lawyers, and theatrical coaches (Ekman, 1966, p. 3). Ekman posited that if these professionals were trained to have a keen eye for nonverbal behavior, so too could other Americans, including those in decision-making positions.

Then what was the advantage of training Americans to have such ability? Ekman provided his answer in the subsequent pages of the proposal.

Americans could then be trained in more accurate communication with people of a given country, whether in the role of military advisors, diplomats, or Peace Corpsmen. Such training would involve teaching the American national what the specific acts of foreign nationals may signify and also teaching him what his own facial expressions and body movements will communicate to them (p. 6).

Here, Ekman was exploring the idea put forth by Osgood and North's theory of the “stimulus-response model of nations,” with which they expanded their insights on individual behaviors to an international scale. In the end, the decision-makers in international relations who would push the button or make the final call were only human. The idea of the human in a human-minded nation in Ekman's proposal was shaped by his double career as a clinical and research psychologist at Langley Porter and as the chairman at the ABSSOP.

The concept of “facial sciences,” as presented by Ekman in the mid-twentieth century United States, parallels the popularization of physiognomy in nineteenth-century London.11 Historian Sharrona Pearl explained that in the rapidly industrializing London, physiognomy offered residents a means “to make quick-and-dirty judgments” to discern whom to trust and whom to avoid (Pearl, 2010, p. 4). In the place of the “industrializing London” was the “culturally expanding America.” Mid-twentieth century Americans had to face unfamiliar faces as the United States superpower was strengthening its international alliances against Communism. American foreign service personnel, who were at the forefront of these intercultural encounters, were in urgent need of systematic training to enhance their communication with individuals from diverse cultural backgrounds. In writing the proposal, Ekman sought advice from Edward T. Hall, a Foreign Service Institute anthropologist who developed training sessions to enhance intercultural communication skills for his diplomat students (Ekman, 1987).12 The proposal reflected Hall's position that a deeper understanding of the nonverbal behaviors of other cultures could reduce intercultural tension in foreign countries (Hall, 1950). Lee Huff, who proposed Ekman to write the ARPA proposal, also had foreign service experience when he served in advisory capacities to the Thai government for several years. Ekman recalled that Huff married a Thai woman and believed that his recent marital problems could have been from cross-cultural nonverbal misunderstandings (Ekman, 2016, p. 45). The increasing demand for facial sciences and face-reading techniques were direct responses to the challenges posed by the expansion of intercultural interactions in various levels of relationships, from international politics to private lives.

With the rationale for a cross-cultural study on facial expressions in place, the remaining question to address was the design of the training program. If one should and could train military advisors, diplomats, or Peace Corpsmen to read faces like psychotherapists, what would the training be like? Ekman offered a glimpse into his thoughts by referring to a pilot study of the classroom environment by psychologist Nathan Maccoby and his team. Maccoby suggested that facial expressions could serve as indicators of how well the students were following what was being taught. The study showed that teachers could be successfully trained to interpret the facial expressions of students, which could contribute to facilitating classroom communication and enhancing students' performances (Ekman, 1966, p. 19; Maccoby et al., 1964). Inspired by Maccoby's work, Ekman also explored whether simple practice could enhance the ability of third-person judges to identify either stress or catharsis from a person's nonverbal behavior in photographs. The subject in the photograph was a junior psychologist who was in a stressful interview situation with a senior psychologist. In the experiment, there were two groups of judges; the first group received feedback from the first set of trials, while the second group merely skimmed through the same set of photos. The experimenters measured the effect of practice by comparing the accuracy of the last set of trials between the first and second groups. The result was inconsistent, and the authors had to conclude that the effect of practice could not be determined by the experiment (Ekman, 1965b). While the results were not very promising, more could be said about the underlying assumptions in these experiments. Both studies involved the analysis of faces, whether they were the faces of students in classrooms or junior psychologists in interviews. Notably, these individuals occupied a subordinate or objectified position in their relationships with the observer-judges, akin to the relationship between the patients and their psychiatrists. The imbalanced power dynamics at play in Ekman's psychiatric film collection and his experiments served as a foundation for his subsequent studies, which began with his cross-cultural nonverbal behavior research.

As Ekman and Osgood delved deeper into the specifics of their collaborations, it became evident that they belonged in two different worlds.13 Osgood was interested in how the subjects respond to the facial expressions of others and sought to comprehend the meaning system of the human face. The stimulus materials included posed facial expressions photographs by actors as well as abstract representations such as the “stick faces” – simple line drawings that relate specific features of the face to emotional labels - which were akin to morphemes of languages (see Figure 4). The human subjects were instructed to rate the stimulus faces, either stick faces or posed facial expressions, based on the pan-cultural semantic differential scales.14 The goal was to analyze the subjects' perception and affective reactions toward these stimulus materials, which were faces bound to interpersonal communication. Osgood's interest had always been in the semantic systems of facial communication, which could be incorporated into their new project, the Atlas of Affective Meanings.

Figure 4

Open in figure viewerPowerPoint

“Stick Faces,” Leon Jakobovits to Paul Ekman, December 2, 1965. (PE Papers, Box 15 Folder: Osgood, Charles, 1965.)

In contrast with Osgood, Ekman maintained a more extractivist view of the human face, understanding it as primarily a source of information.15 He categorized nonverbal behaviors into three types: idiosyncratic acts, gestures, and a third category that fell in between the former two. Ekman explained that Osgood was more inclined toward studying gestures, whereas he himself was more interested in the third category, which included most nonverbal behaviors that “conveys information about the sender, but usually, there are no precise verbal equivalents” (Ekman, 1965a, p. 6). This information included “affect,” which was defined as “both momentary changes in emotion and more durable, characteristic moods” (p. 12).16 Ekman further added that the observer could interpret the meaning of affect information from nonverbal acts, even without knowing the situational context. In other words, facial expressions possessed an inherent meaning on their own, which was independent of both the sociocultural background of the person making the face, the person observing it, and the situation in which the face was performed.

Despite these differences, Ekman and Osgood's partnership continued. Both groups acknowledged the importance of studying facial expressions and the practical financial benefits of their collaborations. After securing the funding from ARPA, Ekman and Osgood sought to bridge the gap between the two worlds and co-organized a “Cross-Cultural Nonverbal Planning Conference” from September 12 to 14, 1966. The conference provided a venue for Ekman and Osgood to share the accomplishments of their recent studies. Osgood delivered a presentation on the meaning system of facial expressions. Another member from Osgood's group again underscored the advantage of “stick faces,” emphasizing their effectiveness as tools for understanding the basic components of the facial language, which would pave the way for a comprehensive understanding of the meaning system of facial expressions. (Center for Comparative Psycholinguistics, 1966).

In contrast, Ekman aimed to work with real human faces, from patients to individuals of different cultures. Ekman's disapproval of stick faces at the Osgood-Ekman conference was stronger than ever, with a clear declaration that they were just “not acceptable” (p. 4). This was because Ekman had grown to become skeptical about utilizing a strict linguistic approach to analyzing faces throughout their planning process. Three months prior to the conference, he attended a presentation by psychiatrist George F. Mahl on “Gestures and Body Movement in Interviews.” In his presentation, Mahl differentiated two distinct approaches to the study of nonverbal behavior. One was the well-established focus on expressive behaviors, such as “rubbing nose” and “scratching self,” attributed to Darwin's pioneering work in the late nineteenth century. The other was the emerging emphasis on communicative behaviors, including “pointing” and “making fist,” which was gaining prominence at the time of Mahl's presentation (Mahl, 1968, p. 304). This distinction between expressive and communicative behaviors allowed Ekman to gain a clearer perspective on his research interest, contrasting with Osgood's preference for the latter, and became the topic of debate during the Osgood-Ekman conference (Center for Comparative Psycholinguistics, 1966, p. 1). Mahl's work ended up accentuating the disparities between Osgood's and Ekman's approaches, which was against the original purpose of their gathering.

Another issue raised during the Osgood-Ekman conference was about their potential field sites. Osgood suggested several locations, including Mexico, Turkey, Japan, and Hong Kong, which were from his cross-cultural studies network. In response, Ekman presented the results from his pilot cross-cultural study, in which he compared the abilities of Mexicans and Americans to judge American facial expressions. The results showed that Mexicans outperformed Americans, and this led Ekman to give two tentative conclusions: either Mexicans are innately better at interpreting emotions from facial expressions, or they had acquired this skill by watching American silent films or interactions with American tourists (p. 3). The takeaway from this experiment was the realization that their cross-cultural research had to take place in a “virgin territory” where there were no movies or television (p. 8). Osgood's field sites were relatively “high” cultures and did not meet Ekman's rigorous criteria for cultural isolation. Given the marked disparities in both methodologies and experimental conditions, the prospect of the Osgood-Ekman collaboration began to appear less optimistic. With the fate of their project remaining uncertain, Ekman began exploring alternative paths.

4 THE “STONE AGE” FACES IN FILMS

In May 1966, just a week after informing Osgood about ARPA's approval for the three-and-a-half-year grant, Ekman reached out to NIH researcher D. Carleton Gajdusek.17 Gajdusek and his cinematographer, E. Richard Sorenson, were known for establishing a film archive at the NIH, which held a positive outlook on the use of ethnographic film for scientific investigations into human events (Link, 2016). Within this collection was an unedited 60,000-foot motion picture film, which documented two cultures in the New Guinea Highlands. Despite sharing similar genetic and ecological conditions, these cultures exhibited distinct approaches to child-rearing and other behavioral norms. More importantly, the filmed subjects were known to have limited exposure to Western culture and its media (Sorenson & Ekman, n.d., p. 3). Each of these conditions was in perfect alignment with Ekman's standards.

After the unsuccessful Osgood-Ekman Nonverbal Planning Conference, Ekman headed to Washington DC to meet Gajdusek and borrow some of the reprints of his films.18 Upon his return to his laboratory, Ekman and his collaborator Wallace Friesen reviewed the films using their slow-motion film projector when a clear shot of facial expressions came into view (Ekman, 2016). Ekman recollected that the expressions captured in the films felt familiar and comprehensible, marking their first “glimmering” realization of the universality of expressions. This moment would eventually evolve into Ekman's theory regarding the universal elements of facial expressions, a theory that he steadfastly maintained for many decades to come (Ekman, 1987, p. 21). Yet, there was another possible explanation for this sense of familiarity. The films were originally taken to include the broader context of each expression and were not subject to further editing. Ekman and Friesen could have been interpreting facial expressions based on what they could infer from the overall situation, rather than just the expressions themselves. Their hypothesis required further verification.

Ekman decided to consult a credible face reader who had no knowledge of the cultural or situational context of the film. From January 22 to February 1, 1967, psychologist Silvan Tomkins visited Ekman's laboratory upon invitation. Tomkins, who developed the “affect theory” that classified the human affective state into eight categories, became a significant influence on Ekman's theories and practices. With Tomkins, Ekman played the edited excerpts of Gajdusek and Sorenson's films that focused on the facial expressions of different cultures in New Guinea. Tomkins carefully observed the eye movements of the subjects and identified fear and shame. In other scenes, he noticed signs of anger from the wrinkles above the nose and mouth. The unusual mouth openings, which moved more vertically than horizontally, suggested the presence of surprise or fear, along with joy (Ekman, 1967b, p. 2). When asked for his overall impression of the two cultures portrayed on film, Tomkins, in Ekman's words, “performed what seem[s] almost an act of magic” in describing their characteristics. He portrayed one group, the Fore, as friendly, while the other group, the Kukukuku, appeared to be explosive in anger, highly paranoid, and have homosexual tendencies (p. 1; Ekman, 1987, p. 21). Ekman was surprised by the accuracy of Tomkins' analyses, which fit perfectly with the descriptions provided by Gajdusek and Sorenson. After spending 11 days with Tomkins, Ekman recognized an immediate improvement in his own ability to read facial expressions, finding it “rather uncanny.”19 This experience bolstered Ekman's confidence about affect being displayed on the face, which he saw as providing substantial evidence to support his earlier hypothesis that the face conveys information by itself.

Ekman also invited Gajdusek and Sorenson to his laboratory in May 1967 to discuss and demonstrate how their films could be analyzed. During their visit, Ekman worked in pairs with Gajdusek or Sorenson, who gave a detailed explanation of the situations in which the films were recorded and their cultural significance. The group played the film multiple times at various speeds and discussed what they saw from the projected faces, which allowed Ekman to validate his “theoretical formulations” (Ekman, 1967a). In return for Gajdusek and Sorenson's films and ethnographic insights, Ekman was to provide a machine system for indexing various nonverbal behaviors in their films: the VID-R system (Visual Information Display and Retrieval system) (Ekman, 1967c). Developed by Ekman and Friesen, the VID-R system featured a custom-built film chain that could project and transfer films at various speeds and zoom in at a maximum ratio of 4:1. This was a significant advancement over the standard film-to-video tape transfer process, which was limited to a fixed ratio of 1:1. With the zoom function, researchers could enlarge specific portions of the filmed image, such as the subject's face, to fill the entire screen. After Ekman's demonstration, Sorenson saw great potential in this system that could transform his films to become more appropriate for scientific research (Sorenson et al., 1967). Anticipating a rosy future ahead of them, Ekman, Gajdusek, and Sorenson embarked on their expedition to New Guinea to document and analyze the facial expressions of the New Guinean people.

5 THE SEARCH FOR THE ISOLATED FACE

From December 1967 to January 1968, Ekman joined Gajdusek and Sorenson in New Guinea, who had arrived earlier for their Kuru research. Traveling from one village to another, Ekman reconfirmed that the Fore culture qualified his criteria of “visual isolation.” Ekman observed that many of the Indigenous people struggled to discern the sex of the caucasian actors in his facial expressions photographs. They even misidentified two female members in the research team as male (Ekman, 1971). The inability to classify the biological sex of Caucasian faces was seen as compelling evidence that the Fore subjects were indeed least acculturated and, therefore, were well-suited to the study of universal human behaviors.

Ekman carried out a series of experiments at Sorenson's field station in the Wanitabi area (Gajdusek, 1969, p. 92).20 One experiment was the “judgment study.” In the experiment, Ekman provided the subjects with a set of facial expressions photographs and instructed them to judge the emotions displayed in each face. The judgment was made by matching each photograph with specific “emotion stories,” which were emotional episodes that the Fore subjects could relate to. For instance, the story of “fear” describes a situation where a person alone in the village without any weapons suddenly encounters a wild pig. The story explicitly stated that it concerned the fear of being bitten to avoid being mistaken as “surprise” or “anger” (Ekman & Friesen, 1971, p. 126). It is worth noting that Ekman used photographs of actual human faces, with actors posing to display specific emotions or test subjects genuinely experiencing various affects, instead of using the symbolic “stick faces” suggested by Osgood.21 His preference for photographs over drawings reflected his belief that the physicality of facial expressions was inscribed with genuine emotions that were beyond any language or consciousness.

Another experiment was the component study, also referred to as “affect portrayals.” The experiment resembled the data-collecting practices at Langley Porter. The researcher narrated one of the “emotion stories” from the previous experiment and instructed the subjects to “show how his face would appear” if they were the person in the story. Ekman recruited nine subjects who did not participate in other experiments, which meant that they had never been exposed to the stimulus photographs. Throughout the entire session, he recorded the behaviors of the subjects, including the moments of listening to the instructions, pausing for reflection, posing to express the specified emotion, showing signs of embarrassment, and interacting with people passing by (Figure 5) (Ekman, 1971, p. 9). Their facial expressions were blended with the embarrassment, confusion, frustration, or nervousness that arose spontaneously in response to the task itself or the comments from their friends who were standing and observing the experiment behind the camera (Ekman & Cole, 1972, p. 275). Ekman's effort to capture facial expressions in a controlled environment was impeded by the impossibility of constructing a completely sterile environment, free from the quotidian interactions in the Fore community. The task of examining the face in isolation turned out to be more challenging than imagined.

Figure 5

Open in figure viewerPowerPoint

Affect Portrayal experiment (D. Carlton Gajdusek Papers, Folder: Sorenson - photographs).

Central to the difficulties of isolating the Fore faces was the disparity between the ideal of a well-controlled psychological laboratory and the reality of an untamed “Stone Age” culture that was resistant to containment and manipulation. Gajduesek and Sorenson foresaw this gap during their year-long preparation for the New Guinea expedition and did not fully embrace Ekman's data practices. Indeed, Ekman was not the first to apply psychological tools to study remote cultures (Lemov, 2011). Nevertheless, his experimental techniques, such as controlling the environment to detach their subjects from everyday social interactions, deviated from the approaches employed by the NIH researchers with decades of experience in the New Guinean cultures.

While Sorenson diplomatically expressed his reservations about Ekman's approach, describing it as a “pioneering attempt to apply sophisticated Western psychometric techniques to non-Western people” in discussing its limitations, Gajdusek was more outspoken about his doubts.22 In one of their meetings before the expedition, Gajdusek shared his concerns about conducting individual interviews with the Fore people.

The very phenomenon of putting anyone in private without anybody else around is a great violation … there is no one in the whole cultural history [who] had ever gone alone, except when he was really alone … And so the very concept of this situation, come in for an interview, no stay out, this is an interview, is such a flagrant violation of the whole life of the village.23

Gajdusek was aware that isolating a subject for experiments and observations was feasible if it was a psychological laboratory. Not being in one, he cautioned Ekman that it would be almost impossible to capture a sole face on film and doubted that their visual recordings could compare to the patient films stored in Ekman's film library. In his diary, Gajdusek wrote in a more candid voice that he was troubled by Ekman's persistence with his method and found it “disconcerting” (Gajdusek, 1969, p. 92). Gajdusek's disappointment with Ekman's approach led him to have more confidence in his own naturalistic and ethnological methods. In the end, Gajdusek began seeking methodological alternatives for analyzing nonverbal behaviors. One such method was “kinesics,” developed by anthropologist Ray L. Birdwhistell in the early 1950s, a method that Ekman had previously expressed his negative opinion to Gajdusek in their earlier conversations (Gajdusek & Sorenson, 1968).24

Another moment of tension unfolded as Gajdusek and Sorenson shared their ethnographic insights with Ekman. Gajdusek and Sorenson emphasized the significance of their research facility, field experience, expertise, and most importantly, close relations with the locals that allowed them to continue their studies in New Guinea (Anderson, 2019). In the process of arranging their collaboration, however, Gajdusek sensed that Ekman regarded him and Sorenson as “mere experimental subjects” in his psychological laboratory.25 Gajdusek and Sorenson had already shared so much with Ekman, including the New Guinea film materials and journals, while Ekman, in comparison, was working alone without consulting them on the details of his experiments. During their expedition, Gajdusek was again disappointed at Ekman's disregard for his and Sorenson's films and expertise as “available information to play with, poach upon, solicit in consolation and problem-solving, but not requiring any further acknowledgment in his formulations than he gives to psychiatric patients or university students of his psychological testing experiments” (Gajdusek, 1969, p. 61). Gajdusek's assessment of Ekman was somewhat accurate, given Ekman's disapproval of the traditional nonverbal communication research for its absence of “empirical evidence” and preference for statistical analysis on the data produced from carefully controlled experiments. The ongoing disputes about prioritizing ethnographic experience or statistical data caused recurring problems that had lasting implications for Ekman and Sorenson's future collaborations. Ekman's aspiration to produce a minimum of six journal articles came to an abrupt end, with only one collaborative essay published. Other findings from the expedition appeared in separate articles (Ekman & Friesen, 1971; Ekman et al., 1969; Sorenson, 1975).26

The strained relationship between Ekman and Sorenson grew more tense as they failed to reach a consensus on their interpretations of the “isolated culture,” a contentious issue that dissolved the Ekman-Osgood alliance. From 1967 to 1968, the previously isolated southern region of New Guinea was going through the early stages of social, economic, and cultural transformations. The growing presence of “Westerners,” including government officers, missionaries, and scientists, meant that some Fore people were beginning to familiarize themselves with “Western” facial expressions and emotional displays. Sorenson saw this as a threat to the integrity of Ekman's research, which was founded on the premise that Fore people were “untouched” by Western cultural influence. Sorenson suggested that they turn their analysis the other way around and use their data to investigate the cultural changes he had been witnessing in recent years (Sorenson & Ekman, n.d., pp. 7–8). Sorenson, who would soon earn a PhD in anthropology and lead the Smithsonian “urgent anthropology” program, felt responsible for exploring the underlying cultural contexts of the experiments conducted at various phases of their project (Link, 2016). Ekman did not. While Ekman conceded that the Fore was going through the stages of acculturation, he contended that it did not affect the results of their experiments. Ekman was confident that the recruited subjects met his “sound criteria” of a sufficient level of isolation, which could be proven by analyzing the data they had already acquired. With the assumption that male subjects were likely to have more encounters with Western culture than female subjects, Ekman used their female data to support the credibility of their male data. His rationale was straightforward: if the results were due to the acculturation of the South Fore rather than the pan-cultural aspect of the facial expressions of emotions, he “wouldn't expect the women to do just as well as the men.”27 This was an idea that required scientific thinking, which, in Ekman's view, Sorenson failed to fully pursue.

The face of the Fore and their face-judging ability were translated into various forms, from moving and still photographs films to numerical data. Over time, these numbers were subjected to rigorous statistical analyses, leading to the initial findings of Ekman's research that underpinned his proposition regarding the existence of universal elements in the facial displays of emotion (Ekman et al., 1969). The visual recordings also took on a life of their own and turned into stimulus materials in Ekman's laboratory after his return from New Guinea. They were also utilized as a means of virtual witnessing to overcome the limited access to his cross-cultural experiments (Shapin & Schaffer, 1985). In The Face of Man and other exhibitions, Ekman used his still photographs and films as vehicles to bring the audience to travel with him through time and space across the Pacific and become eyewitnesses of universal facial expressions of emotions (Hoffman, 2008).

Returning from New Guinea, Ekman further developed the films, still photographs, and data from faces that were successfully dislocated from “Western” influence, interpersonal interactions, and the contexts of their emergence. Throughout this process, Ekman's belief that the face is a promising data source grew into a conviction that there was a universal law in the face of man. This concept resonated across disciplines, extending beyond the realm of psychology, captivating the imagination of the general American public. It was built upon the psychologist's audacious belief that the human face could be isolated and harnessed for scientific exploration into the human.

6 CONCLUSION

What happened after Ekman returned from New Guinea is a story that has become much more familiar to historians of science. Ekman published and further refined his theory of pan-cultural elements in facial expressions of emotions based on the data that he collected from the South Fore (Ekman & Friesen, 1971; Ekman et al., 1969). As sociobiology gained traction in the 1970s, Ekman's sciences began to receive more recognition from scientists and the general public alike. He became a notable public figure, offering consultation to industries, academia, and state surveillance entities. Nevertheless, Ekman's techniques of isolating the face from the whole body and its situational context faced ongoing criticisms from scholars who argued that faces are always communicative, intertwined with social interactions, and are shaped from cultural influence. The debate on the significance of the face in interaction versus the face in isolation has brought the human face to emerge as a promising yet contentious object of scientific inquiry.

Examining the historical context of the making of Ekman's facial sciences and data practices complicates the narrative of a controversial science. Ekman's interest in nonverbal behaviors as sources of information began from his inquiries into human interactions and conflicts. His audacious usage of films to isolate faces from the context of their emergence stemmed from his attempt to overcome what he saw as limitations of his predecessors who studied faces in interactions. Ekman's data practices – which included meticulous control over the filming environment, the reduction of interpersonal interactions to individual expressions, and the compartmentalization of the face and body parts – were shaped and concretized from his collaborations with those who were at odds with his approach and saw facial expressions as being culturally shaped or functioning as communicative channels. Within years, the surface of the face came to be regarded as a physical reflection of our inner selves by those who embraced and expanded Ekman's theories, experiments, or instruments.

The historical shift from the face in interaction to the face in isolation that this essay has explored underscores our understanding of the face as a multilayered entity. The human face encompasses political, ethical, cultural, social, and physical dimensions. Its meanings are inherently diverse, shaped by the interactions between the observer and the observed or the self and the other. It becomes even more complicated with the rise of new sciences and technologies that analyze, categorize, manipulate, and reproduce faces (Pinchevski, 2016; see also Black, 2019, 2023; Pearl, 2017; Plájás, 2023; Watson, 2004). The contrasts between the modern and the old are evident in these developments; while face-aggregating technologies in the past were “still meant for human eyes – specifically, those of the position of privilege and power,” facial recognition technologies nowadays work “out of sight,” perceiving the face “as a particular configuration of surface measurements to be processed analytically and statistically” (Pinchevski, 2016, p. 195). Ekman's science and data practices marked a critical moment in transforming facial analysis to take place “out of sight.” Exploring its history prompts us to reconsider the observer, the human subjects, and the mediating instruments that are involved in this undertaking to gain a deeper understanding of the stakes and consequences of asserting control over human faces. It prompts us to avoid reverting to the “physiognomic tradition” of our troubled past and instead embrace the tension and find the balance between the ambition to isolate and the resistance to isolation.

ACKNOWLEDGMENTS

The author is grateful to the two anonymous JHBS referees for their insights and critiques of the earlier draft. She also thanks Mark White and the staff at the Human Studies Film Archives, Smithsonian Institution, Sarah Harris and the staff at the University of Illinois Archives, Valerie-Ann Lutz at the American Philosophical Society, and Alessandro Pezzati at Penn Museum for their support at the archives. The earlier version of this essay benefitted from the comments by Chihyung Jeon, Heesun Shin, Miryang Kang, Seung Hee Cho, Sungeun Kim, Youjung Shin, and Aaron Gluck-Thaler. The research was generously supported by the Consortium for History of Science, Technology, and Medicine, the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea under Grant No. (NRF-2022S1A5B5A17044801), and the D. Kim Foundation.

CONFLICT OF INTEREST STATEMENT

The author declares no conflict of interest.

ARCHIVAL COLLECTION

Carleton Gajdusek Papers, National Library of Medicine.

Charles E. Osgood Papers 1939-1982, University of Illinois Archives.

D. Carleton Gajdusek Papers Series, American Philosophical Society Library.

Paul Ekman Papers, Human Studies Film Archives, Smithsonian Institution.

Ray L. Birdwhistell Collection, Penn Museum.

ENDNOTES