Between People and Paper: Inhabiting Experiment in a Journal Club

ABSTRACT

In 2015, the Open Science Collaboration reported in the journal Science that a disturbingly large proportion of psychological studies cannot be replicated (Open Science Collaboration, 2015). The ensuing ‘reproducibility crisis’ became a lightning rod for contesting what counts as legitimate research, and for negotiating the relationship between communication infrastructures and research practice. In the psychological and cognitive sciences, the Open Science community has advocated widespread reforms to incentivize transparency, encourage replication, and detect and discourage questionable research practices. The model of ‘openness’ underlying mainstream Open Science centers on sharing information to increase science’s self-correcting capacity. Against the backdrop of broad-scale transformations in Open Science, this case study depicts how scientists read. By examining the activity of a group of researchers ‘virtually witnessing’ an experiment together, this study reveals reading as a non-trivial process that matters for how research is apprehended and for how science is moved through time and space. The case complicates a disembodied, information-centric ‘openness’ pursued by mainstream Open Science reforms and advocates integrating situated and embodied resources into methods reforms, beginning with practices of reading.

KEYWORDS:

• Open science
• experiment
• reading
• cognitive neuroscience
• ethnomethodology
• architecture

Introduction

Opening Acts of Reading

Science is in the midst of refashioning itself. The transformations under the banner of ‘Open Science’ describe a range of efforts to increase the reach, accessibility, and credibility of science, and to change how scientific research is shared, both with broader publics, and between scientific specialists at various stages in the research process. Openness is positioned by leaders in this movement as the revolutionary counterpoint to an older, closed and ‘incredible’ science (Edwards, 2016; NAS, 2018; Nosek et al., 2018; Vazire, 2019). Methods reforms formalizing procedures for how and when scientists share their hypotheses and research/analysis plans, and report their methods, code, and data promise to improve the quality of research by opening closed research and analysis practices toward a more transparent, self-correcting, and credible research culture. Philip Mirowsky has critiqued these promises, locating changes in broad trends toward data-fication and neoliberal platform capitalism (Mirowski, 2018). I identify a related risk of information-centric openness: in defining ‘opening’ science as sharing ever more information, these reforms overlook the situations and processes through which scientists apprehend and interpret such information.

This article is aligned with recent calls for ethnography and qualitative research to explain the failures of replication reforms (Nelson, 2021, Peterson and Panofksy, 2021). Rather than focus on explicit replication practices, I target something more basic - practices of reading that undergird both actual replication, and other forms of peer oversight that are part of Open Science. Against the backdrop of these reforms, I ask: how are reading practices consequential for cognition research, and how can attending to reading practices inform efforts to ‘open’ science? I frame the question by articulating parallels between how canonical STS scholarship and Open Science have configured texts and neglected the role of reading in scientific practice. To answer it, I analyze a social reading practice done by cognitive neuroscientists in a journal club meeting.

A significant cluster of activism and discourse in the methods reform strand of Open Science is found in the psychological and cognitive sciences, though not without parallels in other fields (Nelson et al., 2021). A well-funded and growing ‘metascientific’ social movement centers around Open Science efforts (Peterson and Panofksy, 2020). In this community, transparency reforms are aimed at improving reproducibility and credibility.¹ On the design end, these reforms aim to render explicit scientists’ reasoning and assumptions early in the process, helping to prevent ‘questionable research practices’ (QRPs) like post-hoc analysis (hypothesizing after results are known, or HARKing) and P-hacking (a widespread practice of fudging with the statistical analysis to produce a significant result). On the evaluation end, the aim of these reforms is to make it easier for scientists to check one another’s work, by apprehending it ‘virtually,’ and/or by reproducing all or some of another’s procedures. How scientists form accounts of others’ work through reading is therefore material for reforms and debates around how Open Science might improve scientific credibility and replicability, even though reading as an activity in-itself is largely taken for granted in these conversations.

Of all the experiments a scientist encounters, many more will be experienced through reading than directly. Yet, the remedial opening of science just described calls for the sharing of ever more information, while ignoring the activity by which one scientist (or group of scientists) apprehends and forms an account of (an)others’ work. The case described here will demonstrate that cognitive scientists’ local acts of reading are consequential for the travel and transmission of research designs. This resists an oversimplified picture of ‘openness’ as unfettered access to decontextualized information.

My orientation toward acts of reading as consequential provides a different theoretical framing of inscriptions and their role in scientific practice. Influential accounts in STS have similarly overlooked reading, tending to emphasize the disseminative power of inscriptions as a type of mass media. Shapin and Schaffer, in Leviathan and the Air Pump, (1985) describe how in the eighteenth century, Robert Boyle developed the predecessor of todays’ scientific article that could unite and ‘multiply’ readers across time and space in a shared experience of ‘virtual witnessing’ (Shapin and Schaffer, 1985, p. 60). Virtual witnessing made Boyle’s air pump experiments more accessible for readers who would otherwise would have faced geographical and class barriers to witnessing experiments directly, and supported experimental science's rise to prominence and power.

Bruno Latour, similarly, discusses the way that twentieth century printed inscriptions such as printed maps, charts, and figures consolidated and standardized the power of scientific accounts of the world (Latour, 1986). In characterizing the power of inscriptions in the epistemic economy of laboratory life, Latour and Woolgar describe laboratories as organized around the production and circulation of inscriptions² – a somewhat cynical reduction of knowledge-seeking to the goal of pushing paper (1979). These canonical accounts are helpful for showing how the widespread dissemination of scientific inscriptions bolstered scientific authority. They also contextualize current-day pressure to publish that produce the conditions for fraud and QRPs - major targets of the methods reform strand of Open Science. However, these accounts reach a limit in their ability to consider the pragmatic relationship of inscriptions, particularly accounts of experiments, to the experimental practices that both shape and are shaped by them. Elaborating what would become a well-worn STS formulation of scientific practice, they treat inscription and action as separate and opposed realms. This opposition now pervades STS, as a general theorization of and turn toward practice, configured as what gets left out of written accounts. Greiffenhagen and Sharrock address this opposition in the STS literature as the ‘front and back stages’ of science. They note this formulation’s association with constructivist STS, which foregrounds the messy, tacit, and fallible backstage practices that make possible the written account, which in turn ‘mystifies’ what really goes on (2011, p. 840).

The view Open Science reformers hold of inscription and action is a realist mirror of constructivist STS's: that articles too easily circulate mistruths and disguise, or ‘mystify,’ the questionable research practices that produce them. This community is strongly motivated to demarcate good from bad science, which they aim to engineer through informational openness. My article joins Greiffenhagen and Sharrock's analyses of mathematics teaching in aiming to ‘dissolve the rigidity of the front/back contrast’ (2011, p. 841). In complicating the parallel contrast present in Open Science, I offer a critical complement to information-centric methods reforms.

This article illuminates continuity between scientific inscription and scientific action accomplished in a social practice of reading. I locate this continuity in experiment. Experimental accounts are not automatically transmitted, but actively and collaboratively materialized. In describing how researchers enact others’ past research activity, I develop the analogy of experiments as architectures. This view of experiments becomes visible by treating reading as a situated accomplishment.

The Case: Journal Club at the Brain and Language Lab

My case comes from an ethnographic study of experimental practices in a cognitive neuroscience lab at a large American research university, conducted in 2014. The lab’s research focus is on language and meaning in the brain, and much of the research carried out there uses a particular technology, electroencephalography (EEG). EEG is a non-invasive brain imaging technology that uses an electrode cap to record a subject’s brain activity. In this lab, EEG is used to capture brain activity during experiments designed to study the processing of meaning and language. Tasks include reading, listening to spoken language, and observing videos depicting words, images, and gestures. Some of their research, including the paper being discussed in the case study, focused on perceptual components like the perception of rhythm. My activities as an ethnographer included observing and videotaping experiments and other daily lab activities, participating as a research subject, assisting with the EEG capping process, reading research articles and attending lab meetings. The videotaped interaction in this article comes from one such meeting.

The Analytic Perspectives section situates ‘virtual witnessing’ in a longer history of Open Science, where underspecified ‘virtuality’ of witnessing persists as a neglect of practices of reading. I briefly review research on journal clubs as spaces of enculturation for junior researchers, which complicate the virtuality of ‘virtual witnessing’ by pointing to the importance of the social setting of reading. I then introduce the ethnomethodological approach I take to analyze how the action contained in an article is made inhabitable by its readers.

The empirical core of the article is an interaction during a meeting of the laboratory’s journal club. In the interaction, lab members materialize an experimental design they read about in order to apprehend and manipulate it, using the material and embodied resources brought to the table. Using narration, illustration, and multimodal transcription, I address the central empirical question: how is reading consequential for the transmission and reenactment of cognition experiments? Analysis of excerpts from the interaction will show that:

1. The experiment is materialized as an architecture, which is important for experiments on cognition.

2. How to read cognition experiments, i.e. how to inhabit and apprehend architecturally – requires attunement to a tacit, embodied register, toward which the experienced researchers guide the junior members.

3. Inhabiting an architecture across multiple bodies and enactments provides members insight for interpreting, anticipating, and planning research.

In my conclusion, I reflect on the implications of this account of reading for STS theories of scientific practice and inscription. Finally, I briefly return to the dominant information-centric conception of openness to suggest that reformers make room for and make use of the kinds of practical action described in the case study, advocating a critical complement I call situated openness.

Analytic Perspectives

Virtual Witnessing and Open Science

An earlier moment in the history of Open Science involved the predecessor of the scientific research article that now sits at its center. Shapin and Schaffer describe how, in the eighteenth century, in the early stages of the establishment of a new ‘experimental form of life,’ Robert Boyle’s accounts of his air pump experiments circulated as the ancestor to our modern journal articles, enabling what Shapin and Schaffer call the ‘multiplication’ of ‘virtual witnesses’ (Shapin and Schaffer, 1985, p. 60). Virtual witnessing, in contrast to direct witnessing, meant scientists could virtually observe the experiment without having to be physically present in the highly exclusive space of the laboratory. Shapin and Schaffer draw a comparison to replication, writing: ‘the technology of virtual witnessing was not different in kind to that used for actual replication. One could deploy the same linguistic resources to trigger in the reader’s mind a naturalistic image of the experimental scene’ (60). Widening the accessibility of the witnessing experience was part of how experimental science gained – and multiplied – its footing in a context where experiment was a new and not-yet widely accepted form of knowledge making.

Drawing a line between this earlier ‘opening’ of science and today, Meskus et al. (2018) describe how Open Science uses online platforms, including social media, to further ‘accelerate’ virtual witnessing. They describe a radically distributed, crowdsourced peer review process unfolding on social media which ‘reconfigures the moral economy of science’ (13). They point out pitfalls of accelerated virtual witnessing - dangers of anonymity, rushed accusations of misconduct, and issues of inclusion (19). This repeats a tension that is present in the original account of virtual witnessing – that transmitting ‘experimental scene[s]’ is trivial, except when it isn’t.

In spite of the power of linguistic resources to trigger a ‘naturalistic image’ of the procedure, actually replicating the air pump experiments posed major problems for Boyle’s contemporaries (Shapin and Schaffer, 1985, p. 225–282). That a scientific report can be widely circulated and gain widespread assent without necessarily being replicable prefigures recent and current reproducibility crises. However, the air pump experiments failed to replicate without ultimately losing their status as experimental facts. In contrast to the current focus on harnessing openness to eliminate QRPs (framed as a primary cause of the reproducibility crisis), the air pump’s replication difficulties show that there are other reasons for replication failure, and that a replication failure may not necessarily disprove its finding. The current tendency to interpret replication failures as damaging the credibility of the original findings is normalized by trivializing how experiments move from page to reader. If reading – accessing the past activity of scientists from their written accounts – is not trivial or automatic, implicating the original research through failure to replicate becomes more complicated.

Shapin and Schaffer’s discussion of failures to replicate points to an alternate explanation that complicates their own account of virtual witnessing’s power. They largely attribute the failed air pump replications to failures to transmit tacit knowledge about the design, standardization, and operation of the air pump itself. (225–282). The difficulties in replicating expose a seemingly inherent inability of paper to close the loop between the written account and the embodied and tacit skill of scientists and technicians. Shapin and Schaffer’s accounts of replication difficulties maintain a blurriness around the function of ‘virtuality.’ Theirs is a paradoxical account of what inscriptions can and can’t do: this shows up most clearly as a tension between virtual multiplication of witnesses and material failures to replicate the air pump experiments. The virtuality of experimental accounts was both essential for multiplying witnesses, and, implicitly, a liability when those witnesses could not materially enact the experiment based on written accounts alone.

If we consider that the process of reading might not be transparent or trivial, we can begin to resolve this tension. By turning our attention to the settings and activities of reading, we can respecify the relationship between inscription and action. An unmoored, ungrounded theory of virtuality that foregrounds the text’s capacity to travel, multiply, and accrue power separates the text from the settings and activities of science, including those settings and activities in which reading happens. Examining reading practices grounds ‘virtual witnessing’ and complicates this neat separation.

Journal Clubs as Grounded Contexts for Virtual Witnessing

While mainstream STS scholarship has written more about the capacities of scientific inscriptions than about the practices of reading them,³ educational researchers have centered reading practice in their consideration of the importance of journal clubs in scientific pedagogy. Journal clubs are meetings where a group of scientists gather to present and discuss a research article. Educational researchers describe journal clubs as a ‘signature pedagogy’ in the sciences in general and neuroscience in particular (Schulman, 2005; Golde, 2007). Golde recognizes journal clubs as important spaces for the enculturation of junior researchers, not just by enabling them to ‘keep up with the research,’ but also by modeling ways to present research and evaluate findings, how to discuss and critique, and developing communicative and interdisciplinary capacities by asking researchers to communicate across generational, organizational, laboratory and disciplinary boundaries (Golde, 2007, p. 346).

These pedagogical accounts of reading help to update and situate virtual witnessing, adding an emphasis on the structure and sociality of settings where virtual witnessing occurs. Journal clubs are settings which enroll local resources beyond the literature itself in order to cultivate scientists. This is an insight that begins to embed texts in social settings which are consequential for learning and professionalization.

This article builds on the insights of educational researchers who identified journal clubs as a ‘signature pedagogy’ for the enculturation of scientists. In order to see how these practices of reading are relevant for the transmission of research designs, I look beyond the enculturating activity of reading in a general sense, to concretize the encounter with an experiment that happens in a practice of reading.

Enacting Experiments as Architectures

In order to describe the ‘virtual’ encounter between readers and an experiment in a journal club, I draw on a set of approaches that have had a small but consistent presence in STS – approaches to everyday scientific action developed by conversation analysts, ethnomethodologists, and multimodal semioticians. These approaches to recording, transcribing, and analyzing interaction have helped scholars describe how scientific phenomena are constituted through interactions between people, representations, and technologies, from the optical pulsar (Garfinkel et al., 1981), to gradations of dirt (Goodwin, 1994) to physical ‘domain states’ (Ochs et al., 1996), to the rover-mediated surface of Mars (Vertesi, 2015), to types of brain tissue or activity (Lynch, 1985; Alač, 2011). Whether the practice examined takes place in an observatory, an archaeological dig, a physicists blackboard discussion, a NASA meeting room, or a neuroscience lab, what these accounts share is a concerted attention to the interactional and sequential achievement of the situation. This orientation is compatible with, if not in every case traceable to Harold Garfinkel’s ethnomethodology (Garfinkel, 1967).

What distinguishes work in this vein from more standard STS treatments of scientific practice is a focus on moment-by-moment sequences of action, often emblematized by highly detailed transcriptions. A more substantive distinction, though, is ethnomethodology's particularly grounded troubling of binaries like real/constructed and material/abstract. According to Michael Lynch in his obituary of Harold Garfinkel (Lynch, 2011), instead of a generalized and expansive theory of social construction which casts science as social and therefore artificial/artifactual, ethnomethodology approaches each science as a ‘science of practical action’ in which artifacts and objects are held together in a gestalt figure-ground relationship (Lynch, 2011, p. 935). One of the main contributions of ethnomethodological treatments of scientific practice has been to show how scientific objects (i.e. pulsars, domain states, brain activity) are enacted. The present study builds on this body of research, but finds its gestalt ‘figure’ in the experimental design, which is foregrounded in this interaction.

While ethnomethodology is known for analyzing the practical action found in workplace activities, Lynch points out that its understanding of practical action also includes activities typically distinguished from practice as immaterial, cognitive, or abstract, such as mathematical proofs and even processes of reading (Livingston, 1995; Greiffenhagen and Sharrock, 2011). In disrupting this dualism, ethnomethodological approaches go against the grain of the mainstream turn to practice in STS. Practice-oriented STS scholarship has, often following Latour and Woolgar, characterized scientific practice as what ‘really goes on,’ the ‘mess’ or ‘ragged methods’ which are ‘behind the official accounts of method (which are often clean and reassuring)’ (Law, 2004, p. 18–19). This separation between inscription (clean) and action (mess) maps onto the ‘front and back stages’ of science as characterized in STS (Greiffenhagen and Sharrock, 2011). However, ethnomethodological analyses find interactional orderliness in supposedly ‘messy’ practice. Further, Greiffenhagen and Sharrock’s analysis of how ‘front’ and ‘back’ are accomplished in mathematics teaching show that there is more continuity and complementarity between front and backstage than the caricature of practice as hidden mess may allow for.

Ethnomethodological approaches offer precedents for elaborating a continuity between action and inscription. Such an approach allows me to describe how cognition experiments are forms of scientific practical action – ones I call architectures – whose communication depends on how they are moved between inscribed and enacted states. I treat the inhabiting of experiments as a local, disciplinary form of the human intersubjective capacity to ‘inhabit each other’s actions’ (Goodwin, 2013, p. 15). I offer the analogy of architecture to talk about what materially happens when scientists ‘virtually’ encounter cognition experiments, because architectures are intentionally designed to be inhabited by others. Architectures, like cognition experiments, are designed for someone to be, for something to happen, inside. In examining the situated activity of scientists inhabiting the experiments of others, I show how written accounts of experiments are not purely theoretical, abstract designs, nor are they steps that can be unproblematically re-enacted. Instead, experiments are materialized as flexible, contingent resources for moving methods through time and space.

A Journal Club Meeting

Most Fridays in 2014, members of the lab and interested colleagues met to discuss a research article.⁴ The group included advanced undergraduates, early and late graduate students, postdocs, the PI, and other faculty. The articles were often chosen by the PI but were sometimes suggested by graduate students. As part of the lab’s focus on the psychophysiological components of meaning-making, the group was interested in questions about the perception of meaningful difference, as well as research investigating the sometimes-contested boundary between endogenous (internally generated, sometimes called ‘top-down’) and exogenous (externally generated) processing. One study used EEG to measure a phenomenon that is both understood as more basic than language processing, and which has a long history in psychology – the perception of meter (see Wundt, 1897, p. 142–153).

This topic was of particular interest to one grad student, Cameron,⁵ and was relevant to the experiment he was in the process of piloting, so he was the one presenting the article. The paper was ‘Top-Down Control of Rhythm Perception Modulates Early Auditory Responses,’ (Iversen et al., 2009). The experiment investigated differences in evoked brain activity in relation to rhythms that were metrically organized from without or from within: that is, subjects heard a rhythm which contained a physically accented downbeat, or they heard a rhythm for which there was no physical accent, and they were asked to imagine the downbeat. The ‘real’ versus ‘imagined’ stimuli conditions, as well as the two accent patterns used, were the main topic of conversation in the interaction (Figure 1).

Figure 1. The meeting/journal club. Clockwise from Cameron, around one long side of the table, are Greg (another new grad student) Mary (a more senior grad student), and Penny (the PI). Across the table from Cameron, at the other short end, is Victoria, a professor with specific expertise in the area of music cognition, who is visiting from another lab. Along the other long side are Rita (another grad student) and Sarah (myself). Penny, Mary and Rita have laptops in front of them on the table; Greg has a tablet. They appear to be using the laptops and tablet to view and scroll through the article that Cameron is presenting.

In these excerpts, Cameron (in collaboration with others present) attempts to convey the stimuli conditions. The transcript tries to capture how Cameron and the lab members are visually, verbally, and aurally available to one another, by combining conversation analytic conventions from Jefferson (2004), with some conventions for transcribing gesture from Mondada (2019) and gaze from Hindmarsh and Heath (2002) (See glossary in Appendix 1). I treated relevant non-verbal sounds, such as demonstrative banging and tapping, as though they are utterances, because they are treated that way by members.

Overview of the Scene:

As Cameron moves into his explanation of the stimuli, he has just been asked by Penny to ‘tap out the stimuli’ so ‘we can hear ‘the way that it sound[s].’ Cameron then adds to his exposition a combination of novel, semi-iconic hand gestures and finger taps (Figure 3). He does this until line 13, when Victoria asks him to continue ‘for several measures’ so that we can hear ‘what the people are hearing.’ Cameron continues to explain while combining hand/finger gestures with brief, accented tapping the table (Figures 4 and 5), but does not continue for several measures as requested by Victoria. After Mary poses a clarification question about a finding, Victoria begins loudly banging out the two accent patterns on the table, during which the conversation between Cameron and Mary continues at the same time as the other members orient their comments toward the ongoing rhythmic sounds. After this rhythmic interlude ends, members continue to discuss the significance of a finding about which condition is easier or harder for subjects, while intermittently bobbing their heads and moving their bodies to a now-imagined, now-embodied rhythm. Some utterances occurred between transcribed excerpts, and are assigned line numbers according to their place in the sequence.

I organize the analysis into three acts, each centered on a different layer of this interaction, foregrounding a different feature of how the experiment is read by the group. The first layer of this interaction (Act 1) centers on Cameron and how he reads by conveying the experimental architecture in multiple combinations of talk, sound, and gesture. The second layer of the interaction (Act 2) centers on moments where the senior researchers guide Cameron and the group in how to read towards inhabiting and how to attend to that reading with their bodies. The third layer of the interaction (Act 3) centers on how inhabiting the experiment provided members practical insight for understanding the paper’s results, as well as for planning present and future research.

Act 1: Apprehending the experimental design as architecture through gesture and sound

Cameron is introducing the experiment described in Iversen et al., referencing the choice between a ‘perceived beat’ versus a ‘physical beat,’ when the PI, Penny, asks whether he wants to ‘tap out’ the stimuli, ‘the way that it should sound.’ In response to Penny’s request to tap out the stimuli, Cameron introduces the repeating rhythmic phrase, or ‘A’ at the top of the diagram in Figure 2.

Figure 2. Reproduces (with permission from the publisher) the figure from Iversen et al. (2009) that Cameron is discussing and that several members have in front of them on paper, computer, and tablet. It lays out the 4 stimuli conditions, which were made from a repeating rhythmic phrase with two accent patterns: first tone and second tone accent, and two conditions for perceiving these patterns: imagined and physical.

This excerpt contains the first time in the meeting that sound is used to convey the experimental stimuli. In this first attempt, Cameron’s demonstration is relatively unannotated and simple. Cameron marks the rests (0.5 second pauses in line 3) by raising his right-hand index finger in a slightly exaggerated way as though to draw attention to the empty beat (rest). In excerpt 1.1, Cameron combines embodied and local resources – his speech, hands, fingers, the table – to introduce the ‘tee-tee-zero rhythm.’

The next time Cameron materializes the stimuli, he has just responded to a clarification question posed by Victoria, ‘They being Iversen et al?’ (line 4) by offering an explanation of what he will do: ‘I’ll describe the stimuli and then what they’re doing because it’s sort of a lot easier to understand once you have the actual stimuli’ (line 5) (lines 4 and 5 occur between excerpts). He goes on to explain that he is describing the ‘unaccented’ pattern, clarifying that ‘the actual stimuli’ means the unaccented rhythmic components, while ‘and then what they’re doing’ may mean the different conditions ‘they’ (the experimenters) manipulate.

In this excerpt, Cameron performs a stylized, repetitive gesture where the rhythmic tones are mapped both onto the tapping sounds and onto the counting down gestures of his left hand. This seems to function to set up, spatially and verbally, that the rhythmic pattern consists of groupings of 3: two tones and a rest (line 9; Figure 3). Cameron emphasizes the distinction between rests and tones, by saying ‘and then’ when the pattern switches between tones and rests.

Figure 3. Line 9. Demonstrating the T-T-0 pattern. Counting down gesture (left hand) with non-accented tapping (right hand).

Victoria, echoing Penny’s initial request to tap out the stimuli, asks Cameron to tap them for longer (‘several measures’) so that we can hear ‘what the people are hearing’(line 13).

Meeting Victoria and Penny’s requests is tricky – ‘what the people are hearing’ in the experiment depends first on which condition you are discussing, and, in the imagined conditions, how you define ‘hearing’ – is it what physically travels to their ears, or is it what they experience between their ears, based on the pattern they’ve been primed to hear by the induction sequence? Cameron’s first two attempts (1.1 and 1.2) consisted of trying to convey the unaccented pattern, or ‘A,’ the ‘repeating rhythmic phrase’ which underlies all four conditions. This ‘basic’ form is the unaccented pattern that the experimenters manipulate, but it is also what is literally pumped into subjects’ ears as they are asked to imagine one or the other rhythm pattern in the ‘imagined’ conditions (IB1 and IB2 in Figure 2). Cameron’s materializations had so far not introduced any sense-able differences that would demonstrate to the group ‘what they’re doing,’ whether ‘they’ refers to the scientists or the participants. As we will see in more detail in the following act, Penny and Victoria, the more senior members at Journal Club, imply that there is a preferred way to perform these experimental conditions, which involves making available a sense-able difference between the physical accent conditions (C).

Finally, in response to Victoria’s request (line 13), Cameron makes a third attempt, one which again begins with ‘A,’ before introducing ‘B’ and ‘C’ in the diagram. Cameron delineates between the ‘basic’ underlying form and the imagined and physical conditions, then verbally introduces the idea of two accent patterns, then audibly performs the two accent patterns for the first time.

In this excerpt, Cameron finally materializes two audibly different accent patterns. However, contrary to what Penny and Victoria seem to be requesting, he gets there by repeating the sequence of the diagram, from most abstract (‘A,’ the T-T-0 pattern) to concrete (‘C’, the physically accented pattern). During the first part of this explanation, Cameron refers back to ‘the basic form of it … the tone tone blank’ (lines 15–16). Then, he delineates the two ‘basic conditions’ – the ‘imagined’ condition, and the ‘physical condition’ (line 19). Cameron proceeds to introduce the idea of the two different accent patterns. He does so first with a phrase and gesture that seem to reference the job of experimental design and the researcher’s agency: ‘the idea is you can choose any of those two actual tones to accent’ (line 21). This is accompanied by a gesture which places the ‘tones’ into Cameron’s upheld left palm, and the ‘choosing’ between which tones to accent is mirrored by the right hand moving back and forth to touch two spots on the left palm (Figure 4).

In Figures 4 and 5, Cameron holds the components of the stimuli in the palm of his hand. His utterance in line 21, ‘the idea is you can choose any of those two actual tones to accent,’ is followed by a demonstration of ‘what the people hear’ that materializes for the first time a physical, sense-able difference – the tapping in lines 22 and 24. With his right hand, Cameron taps out the two accent patterns that ‘the people hear,’ while his left hand continues to be cupped upwards (Figure 5). The right hand demonstrates the accent conditions, ‘what the people hear’ (or imagine they hear). His upheld left hand continues to reference the ‘choice’ of which tone to accent, a choice that could be attributed to the researchers in designing the stimuli, or to subjects in performing the direction to imagine a particular accent pattern.

Figure 4. ‘you can choose any of those two actual tones to accent’ (line 21).

Figure 5. C’s Left hand holds manipulable tones, Right hand TAPS ‘what the people hear’(lines 22–24).

Through this sequence (1.1, 1.2, 1.3), Cameron elaborates the abstract, ‘basic’ version of the experimental stimuli alongside its more concrete, sense-able counterpart. This compresses experimenter and participant as well as imagined and real conditions. Mapping the tones onto his left palm and touching them with his right fingers, he references the ‘choice’ of which tones to accent, a choice which could ambiguously apply to both the experimenter or the subject, the architect or the inhabitant of the structure under discussion. This ‘choice’ collapses the distinction between the work of experimental design, and the cognitive and perceptual labor of the experimental subject that is under investigation, and thus imaginatively projected by that design. His use of an unspecified ‘you’ contributes to this ambiguity (‘You can choose any of those actual tones to accent’ (line 21)). By dividing them between his left and right hands, Cameron simultaneously materializes the positions of experimenter and subject and the experimental conditions they implement or experience. Cameron goes in order, but his hands try to grasp the whole experiment at once, as an architecture that can be apprehended from the outside and the inside.

The architectural duality of inside and outside is an important part of designing experiments in cognitive science, because cognitive phenomena unfold inside of people. Imaginatively occupying the position of subject is necessary for designing experiments that stabilize cognitive phenomena. This imaginative substitution is one part of what constitutes mastery of experimental architecture, or becoming competent at understanding, designing, and communicating experiments (to participants, to other scientists, and to other nonexperts). This, I believe, is what is behind Penny and Victoria’s repeated requests that Cameron ‘tap out the stimuli,’ provide ‘what the people are hearing,’ and ‘do that for several measures.’ In his attempts to present the experiment, Cameron may not fully meet his mentors’ requests, but he nonetheless grasps toward mastery in trying to materialize the experimental architecture from all angles and positions at once. As I will argue in the following section, what the senior scientists do in this meeting of journal club is not to correct any substantive misunderstanding of the experiment, but to guide the group in how to read it together.

Act 2: Doing Reading as Embodied Inhabiting

On one level this meeting of journal club is an encounter between novices and experts. We saw in the previous section how Penny the PI and Victoria the visiting colleague make several suggestions guiding Cameron’s presentation of the paper by Iversen et al. Cameron is both the leader of the meeting and a learner in this context. As the literature on journal clubs shows, what he and other students are learning in journal club includes but also exceeds the content of the text. Similarly, in this case, what is important is not how the experts are imparting informational content to the novices, but how they are working to coordinate resources in the setting: members’ action and attention as the text is read. This section will show how the meeting confers not only tacit (that is, not explicated) knowledge about how to present, discuss, and critique like a professional scientist (Schulman, 2005; Golde, 2007), but also cultivates tacit skill in how to read about experiments. This tacit skill in reading cognition experiments is not an unspoken norm, like how long to speak before pausing, nor is it a hyper-specific embodied skill, like how to ride a bicycle or suture an incision. Rather, it is an attunement to the experiential and tacit dimensions of an experiment that cannot easily be explicated, but can be felt. Specifically, this section shows how the body can act as a resource for reading cognition experiments.

In the excerpts in the previous act, Penny Victoria made several suggestions guiding how Cameron brings what is on the page to the table. These suggestions, in effect, asked Cameron to present the stimuli in reverse order by starting with physically sense-able accent patterns and allowing the group to inhabit the experiment on the way to apprehending it. Going in reverse order from the text may be counter-intuitive for the novice reader, and what Cameron did by holding onto both the inside and outside at once may be a novice’s strategy. The more experienced researchers seem to intuit that inhabiting the experiment first will make its apprehension and, later, manipulation, more possible. In addition to encouraging a non-sequential order for presenting what is on the page, they also coordinate the whole group’s attention toward the sense-able and embodied register of these materializations.

Immediately following Cameron’s demonstration (line 24), Penny gathers the group’s attention to the audible difference between the two accent patterns by saying ‘And they do sound really different, yeah?’ (line 25) and nodding first to the members across the table from her (Rita and Sarah) and then to those to her right (Mary and Greg). Each member of the group directs their gaze to Penny as she is speaking and returns a nod at the end of her question:

Penny’s gathering of the group’s attention bookends her initial request to Cameron to ‘tap out the stimuli.’ Both Penny and Victoria, who are the more senior researchers, had encouraged Cameron to demonstrate the sense-able difference between the stimuli conditions, and here Penny takes advantage of the first time he meets their requests by directing members’ attention to how they ‘do sound really different.’ While Penny’s directing of the group’s attention to the audible difference suggests that she is satisfied with Cameron’s demonstration, Victoria did not have her request for a longer duration met. In the sequence that follows, Victoria performs percussive rhythm patterns simultaneously with the continuing talk. Her extended rhythmic performance responds to, acts as context for, and ultimately becomes an explicit object of discussion around the relative ‘difficulty’ of the stimulus conditions.⁶

In this interlude, the experiment is again performed as an architecture, with positions inside and outside the experiment layered across multiple members’ enacted resources. As Victoria’s percussive rhythms move from background to foreground, they meet earlier requests to make an inside position available by ‘tap[ping] out the stimuli,’ and ‘do[ing] that for several measures.’ Members are also working to clarify the finding Cameron reports that ‘in fact one of them they find is a lot more difficult than the other’ (line 27). In relation to the reported relative difficulty of the stimulus conditions, Victoria’s demonstration makes the accent patterns available for members to calibrate which pattern ‘feels more difficult.’

These background sounds are brought to the foreground as a resource for sense-able calibration when Penny, who has been shifting her gaze between Cameron and Mary as they discuss the difficulty question, turns to Victoria and says, ‘I feel like that would be easier’ (line 40). The percussive sounds are now an explicitly referenced object to which members orient their attention. Victoria incorporates her next rhythmic demonstration into her utterance, saying ‘I think it’s not as easy as’ (line 44) and switching to the second-beat-accent pattern. Penny and Victoria use language of feeling and thinking as they orient to the rhythm patterns: ‘I feel like’ and ‘I think it’s not as easy as’ (lines 40 and 44). In using this language, Penny and Victoria highlight the accent patterns as perceptual objects. They assess and discuss the stimuli by adopting the point of view of the imagined experimental subject. After Victoria says ‘I think it’s not as easy as’ and demonstrates the second pattern, all members move their gaze to Victoria. When she has all members’ visual attention (lines 44–45) Victoria modifies the way she performs the second accent pattern: while hitting the table in the same manner as before with her right hand, she also pats her chest in the same accent pattern with her left hand (lines 45–47). Victoria visibly performs feeling the accent pattern by performing it on her own body as well as on the table (‘tBAM’).

In verbally and bodily orienting to the sounds from the perspective of an experiencing subject, the senior members Penny and Victoria publicly demonstrate getting inside the experiment. These public demonstrations function as an invitation for members to read the experiment this way themselves. In the moments following this final part of transcribed interaction, several members can be seen quietly embodying the rhythm pattern on and for ourselves. Rita taps/moves her right fingers, crossed over her left arm; Greg bobs his head and upper body while looking at his tablet, and Sarah (myself) taps her pen on her lips. Like Victoria’s percussive banging, these actions unfold in the background of the conversation, but they don’t appear outwardly directed as accessible resources for others to act upon. These movements were virtually private – done without exchanging eye contact, and with the exception of Greg’s rocking, barely noticeable on video, let alone to one another. While nearly-private, these self-stimulations are a small, yet observable culmination of this social practice of reading, as iterations of experimental architecture move from the page, into the shared space of interaction, and into members’ own bodies. Members feel the difference between the two patterns, and thus ‘virtually’ reproduce the task the experiment asks its subjects to undertake (Figure 6).

Figure 6. Rhythm patterns, privately embodied.

Penny and Victoria communicate tacit knowledge, but it is not tacit knowledge about a material technology, ie, how to succeed in using a finicky scientific instrument like the air pump (Shapin and Schaffer, 1985) or the EEG machine (Klein, 2014). Instead, it is at once tacit knowledge about how to read, and active attunement to embodied dimensions of reading. Penny and Victoria invite members to read the experiment as architecture by making the tacit, embodied register more intersubjectively available, and the experiment in question more inhabitable. The interaction demonstrates how the body can be a resource for the ‘virtual witnessing’ of cognition experiments from the inside out.

Act 3: Moving methods – inhabiting toward interpretation, planning, and critical encounters

The previous acts showed that the experiment can be materialized and inhabited as an architecture on the way to apprehending its design. But apprehending the experimental design is not the end-point of this activity. The reading practice is also mobilized toward moving methods through time and space, grounding practical insight that can help interpret findings, justify research questions, anticipate problems, and question paradigms. In other words, the interactions that unfold in a social practice of reading are themselves resources for both looking backward and moving forward in science.

We saw in the last act how the question about the relative difficulty of the stimulus conditions became an opportunity for Victoria and Penny to demonstrate the sense-able differences between the rhythm patterns and invite those assembled to feel these differences using their bodies. At the same time as offering up the experiment as architecture so that its design can be grasped, the same inhabiting process enabled members to calibrate the difficulty between conditions and interpret the finding that ‘in fact one of them [the conditions] they find is a lot more difficult than the other’ (line 27). After the interaction presented above, the conversation (not transcribed) turns to positing cultural explanations for the participants’ difficulty in holding one of the rhythm patterns. Cameron and Victoria discuss the suggestion that the difficulty can be explained by culture – that participants used to a Western music idiom are more likely to be habituated to the second beat accent than the first beat accent pattern, which is more common in some non-western music traditions. We also learn that Cameron will pursue this hypothesis in his current and future research. Here, we can see the methods moving from the past, as past research action, into the present and future research action of Cameron’s pilot study. The difference that anchors Penny and Victoria’s assessments about which pattern is easier or harder (lines 40, 44) helps to ground, in Cameron's words, ‘why our research later is interesting’ (lines 46–47). The group, invited to calibrate the difficulty using their own bodies, can use the same experience of inhabiting to anchor and justify a new question, one not addressed in the study they read. The experimental architecture, once inhabited, can be copied, but also manipulated, moved, and adapted.

The process of inhabiting the experiment also offers practical insight into how bodies may behave in the experiment. For example, as part of the subsequent discussion of Cameron’s pilot study, we learn that the pilot data was full of noise and artifacts – distortions in the EEG from sources other than the brain. Typically, with EEG, artifacts are caused by muscular activity which interfere with the measurement of brainwaves. In an EEG study like the one under discussion, the participant is expected to hold still, and may be asked to control their blinks, swallowing, and eye movements to a certain degree (Joyce et al., 2004; Klein, 2014). Initially, Cameron suggested that the artifacts in his pilot data could be eye movements. A suggestion was made next by Sarah (me) that the participants might be moving or bobbing their heads to keep the beat, as several of us had just found ourselves doing. The process of reading as inhabiting served as a kind of soft piloting of the experiment. Cameron could anticipate that participants, too, move their bodies as they attempt to ‘feel’ the beat. Thus, the reading practices I’ve described here are not only an academic matter of understanding the experiment. What happens at that table can itself become material for interpreting, anticipating and planning. It could inform decisions about whether or how to build on the study being read, whether or how to replicate it, or, as in this case, how to interpret one’s own data.

Reading as inhabiting anticipates and rehearses what bodies may do in the apparatus, and this offers the kind of technical troubleshooting we saw emerge in the meeting itself. It also holds potential for a more substantive critical encounter with cognitive constructs and how they are operationalized in experiments. In this case, the process of reading the experiment could shed critical light on the distinction between ‘imagined’ and ‘physical’ as operationalized in the experiment. After all, if participants are inclined, like the group members were, to move their bodies in some way to ‘imagine’ (or apprehend) the accent pattern, was the experiment truly testing an ‘imagined’ condition? If not, what was being measured there? Is it useful to define ‘imagination’ against the physical or embodied? If so, does it make sense to operationalize ‘imagining’ through the particular bodily performance entailed by the EEG? If not, what can EEG be useful for answering? The reading activity performed an unexpected tension between how ‘real’ and ‘imagined’ are premised in the paper and how they were materialized at the table. This far exceeds and respecifies what would be expected from traditional accounts of ‘virtual witnessing,’ insofar as virtual witnessing has been positively configured as an automatic, one-way transmission of experimental accounts, or negatively configured as separate from tacit, embodied and material knowledge. This case shows that it is neither automatic, nor separate from the material and tacit, but is consequential precisely because of the specific combinations and sequences of local resources enacted between people and paper.

Conclusion: Toward a Situated Openness

The case demonstrates how reading can be consequential for the reception and reenactment of cognition experiments. What Shapin and Schaffer called ‘virtual witnessing’ is not mere transmission of content, nor a transparent window onto past research. In grounding virtual witnessing as a situated and embodied practice, this analysis complicates the assumed automaticity and transparency of scientific accounts.

For the cognition experiment in particular, its architectural character becomes visible in a social practice of reading. A cognition experiment is a structure that can be both inhabited/experienced from the inside, and manipulated from the outside, and both positions are necessary for a practitioner to occupy. The case also showed that how to read (about) experiments is a tacit skill in itself, one that was encouraged in a social practice of reading. This further blurred the distinction between scientific inscription and action, where reading is typically figured as a disembodied practice, and the tacit canonically lives between fickle instruments/organisms/materials and the skilled hands of technicians. So-called ‘virtual’ encounters with experiments are not assured, but involve skills that are enacted between people and paper. Finally, the case shows that what emerges between people and paper, whether it be a miscommunication, a contradiction, or a self-directed calibration, can become a resource for interpretation, planning, and critique moving forward.

This has implications for the kinds of reforms to research and norms of transparency being advanced under Open Science. This research does not suggest that more sharing is always bad, or that it is pointless to try to improve transparency. It does suggest, however, that a focus on information-centric openness that neglects how people actually encounter and apprehend research accounts is, at best, a lost opportunity to look inward, and at worst, an instrumentalizing move that lacks epistemic humility. If, as I argue, reading is non-trivial, then improving science is more complicated than requiring more information to surveil more parts of the research process.

As a critical complement to an ideal of ‘openness’ that is defined and constrained by a reductive account of information transmission, I propose attending to, and further, harnessing the kind of situated, or ethnomethodological openness that the researchers in my case study demonstrated. In inhabiting the past action of other scientists, they pragmatically and materially explored the research design they were reading about. In this case, their architectural enactment of reading brought forward an embodied, soft piloting of the experiment. This enactment made available and actionable possible technical problems, as well as inherent conceptual tensions in the operationalization of the experiment's constructs. Taking these enactments seriously as actionable insights is not something that the current reforms make room for. Openness-as-inhabiting complicates the suspicious epistemology that reduces replication to quality control. If we attend to them as inhabitable and enacted, the richness of what replication (and re-enactment, and reading) can mean expands beyond a positive or negative credibility signal.

Situated, ethnomethodological openness to practice and context already lives in the daily activity of scientists, so is not hard to come by. However, the potential of this kind of openness to provide meaningful insights for scientists is in tension with the kinds of narratives about scientific self-correction and communication that dominate OS and methods reform projects. Taking these forms of practical action seriously would mean designing platforms and formats in ways that are flexible and sensitive to how people actually read and practice science; and mitigating the unmoored, universalist, and suspicious projects of information-centric openness with a disposition toward reflexive inhabiting.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Social Sciences and Humanities Research Council of Canada [grant number SSHRC Doctoral Award # 752-2011-0551].

Notes on contributors

Sarah Klein

Sarah Klein is an assistant professor in Communication Arts at the University of Waterloo. Her research on experimental practice in cognitive science has been published in Performance Research and Performance Matters. Her work centers on how methods move, treating experiments as performances with material, embodied, situated, and esthetic features indivisible from their empirical structure. Bringing together approaches from Communication Studies, Science Studies, and Performance studies, Sarah’s collaborative work involves opening up research to performative intervention by documenting art-science collaborations and by designing experiments/performances with scientists.

Notes

1 There are criticisms of OS which target the one-size fits all and top-down implementation of many of these reforms. Some critique these reforms on the grounds that scientific inference cannot be proceduralized (Rich et al., 2021), that top-down reforms harm epistemic diversity both within these fields (Devezer et al., 2019) and outside of them (Penders et al., 2019), that these reform movements are exclusionary (Whitaker and Guest, 2020). These debates are beyond the scope of this paper, but I note that the attention I pay to context-embedded practices of reading resonates with pushback against moves to proceduralization and my account of how attending to social reading practices can reveal implicit theoretical tensions resonates with calls for formal modelling to make hidden assumptions explicit (Guest and Martin, 2021).

2 Latour and Woolgar (1979) develop a capacious theory of inscription and inscription devices that includes any and all kinds of technological trace-making, charts, maps, visualizations, as well as the production and circulation of texts (articles, reports) which may assemble some of these inscribed traces with written descriptions of methods. Because the scale of inscription and what is inscribed varies, it is worth clarifying that in this article, the type of inscription that I center on is the research article itself, and the ways in which experimental designs are moved between inscribed and enacted states.

3 Exceptions outside the mainstream include anthropological work on individual practices of reading (Livingston, 1995; Roth, 2010). A small body of work on historical replication are among the closest to a situated examination of reading practices because they highlight the embodied work of reader/replicators filling in gaps in methods descriptions (Heering, 2008, 2010).

4 I asked if I could attend these meetings as part of my ethnographic observation and was granted permission; at the first few meetings of the school year I introduced myself to the group (a few knew me already) as an ethnographer studying cognitive scientific practice. After a few meetings, I asked whether I could videotape. From then on, when I was able to attend, I would set up a tripod and camera to record the meeting. I participated in discussion, even though my background did not match those of the other students.

5 All names are pseudonyms, with the exception of my own.

6 In the exchange that happened between the transcribed excerpts, Mary asked Cameron to clarify his line 27 utterance, asking ‘what’s more difficult?’ (line 32) and (lines 28–33 occur between excerpt 2.1 and 2.2)

Appendix 1.

Glossary of Transcription conventions

“Unless noted, conventions are adopted from Jefferson (2004).”

Table