![Sample our Law journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5942/TOC-Subject-Sample-2021-Law.jpg"})
Abstract
We all accept that peer review is an essential part of journal publication in all disciplines, but almost everyone is agreed that it could be improved. This article describes an experiment in peer review with a legal education journal The Law Teacher. It reports on the process, and describes ways in which the process can be improved for the future.
Notes
1 S Harnad ‘The invisible hand of peer review’ Exploit Interactive, 5, 2000, http://www.exploit-lib.org/issue5/peer-review
2 For some disciplines, such as medicine, the review process is subject to on-going critique—see for example the International Congress on Peer Review and Biomedical Publication, held every 4 years, at http://www.ama-assn.org/public/peer/peerhome.htm. For an analysis of the processes of scientific peer review (in the field of Chemistry), see H-D Daniel Guardians of Science: Fairness and Reliability of Peer Review VCH, Weinheim, 1993.
3 See M J Mahoney ‘Open exchange and epistemic progress’ American Psychologist Vol 40, pp 29–39, 1985.
4 See for example S Schroter, N Black, S Evans, J Carpenter, F Godlee and R Smith ‘Effects of training on quality of peer review: randomised controlled trial’ British Medical Journal, Vol 328, pp. 673–678, No 7411, 2004, where the conclusion was that short training packages have only a slight impact on the quality of peer review. The value of longer interventions still needs to be assessed.
5 See, for instance, D Rennie ‘Freedom and responsibility in medical publication: setting the balance right’ Journal of the American Medical Association Vol 280, pp 300–302, 1998.
6 See for example F Godlee, C R Gale and C N Martyn ‘Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial’ Journal of the American Medical Association, Vol 280, pp 237–240, 1998. In this study the authors modified a paper accepted for publication introducing 8 areas of weakness. Reviewers were randomly allocated to five groups. Groups 1 and 2 received manuscripts from which the authors' names and affiliations had been removed, while groups 3 and 4 were aware of the authors' identities. Groups 1 and 3 were asked to sign their reports, while groups 2 and 4 were asked to return their reports unsigned. The fifth group was sent the paper in the usual manner of the journal, with authors' identities revealed and a request to comment anonymously. Of all five groups, the median number of errors spotted was two, nobody spotted more than five and 16% did not discover any. The authors concluded that neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports.
7 Peer comment can be helpful and rigorous; but it may not always be so. The most developed commentary on this is probably that of Stevan Harnad, quoted above, and also his numerous contributions to the debate on discussions boards. See the Harnad e-print archives at http://www.ecs.soton.ac.uk/∼harnad/. The process of developing the Wikipedia—an example of social computing in action—may at first glance be a model of peer review; but there are significant differences. A wiki is really an example of an open community of writers adding and editing text indefinitely, and creating a blend of voices. In addition, writers are exhorted to take a neutral and descriptive stand on issues. They also describe the physical and conceptual world, rather than adding to the sum of knowledge within it.
8 See for example, S Hitchcock, L Carr, Z Jiao, D Bergmark, W Hall, C Lagoze & S Harnad ‘Developing services for open eprint archives: globalisation, integration and the impact of links’ Proceedings of the 5th ACM Conference on Digital Libraries, San Antonio, Texas, June 2000, at http://www.cogsci.soton.ac.uk/∼harnad/Papers/Harnad/harnad00.acm.htm. For an experiment from the British Medical Journal in such online peer review, see British Medical Journal, web site, http://www.bmj.com/cgi/shtml/misc/peer/index.shtml.
9 See for example Harnad, op cit, note 1. For a successful example of Harnad's vision, see Paul Ginsparg's work on a global preprint archive, on which the world high energy physics community relies, at http://arxiv.org/corr/home.
10 For an example of a wiki, see the online encyclopaedia, wikipedia, at http://en.wikipedia.org/wiki/Main_Page.
11 Journal of Interactive Media in Education, http://www-jime.open.ac.uk/index.html. We should point out that the review processes of this journal came to our attention only after the experiment described in this paper.
12 See F W Lancaster ‘The paperless society revisited’ American Libraries Vol 16, No 8, pp 553–555, 1985. Lancaster's taxonomy is as follows: 1 computers used for print production 2 journal distributed in both print and electronic formats 3 publication design is rooted in print, but articles are developed solely for electronic distribution 4 interaction between authors and readers is possible; publications can evolve as a result of such interactions 5 the inclusion of multimedia content 6 both interactive participation and multimedia capabilities are supported. Quoted in S Buckingham Shum and T Shumner ‘JIME: An Interactive Journal for Interactive Media,’ First Monday Vol 6, No 2, 2001.
13 See http://www.sweetandmaxwell.co.uk/index.html, where editorials and contents listings can be found.
14 See http://www.lawteacher.ac.uk.
15 Guidelines regarding review transparency are given by the Committee on Publication Ethics, at www.publicationethics.org.uk. The Committee's advice and recommendations on author dispute, for example, are very useful—see http://www.publicationethics.org.uk/cases/zerozeroseven.
16 Although the website was hosted at a public site, its status as a private forum was essential to its character. We comment on the effect of this below.
17 The editors of the special edition also intended to post papers to a public website post-publication in The Law Teacher, but this did not occur. The Law Teacher currently does not exist in a web format, and the discussion forum was part of the Glasgow Graduate School of Law website. While this was appropriate for a private discussion forum area, the special edition editors did not feel, in retrospect, that authors would have felt comfortable about final articles being posted on the same publicly-accessible site as their draft papers and discussion – though they did not collect opinions on this so the matter remains open.
18 N Burbules Dialogue in Teaching: Theory and Practice, Teachers College Press, University of Columbia, 1993.
19 See for instance V M J Robinson ‘Dialogue needs a point and purpose’ Educational Theory Vol 45, No 2, pp 1–17, 1993. Robinson argues that Burbules requires to prove how ‘our normative models of dialogue can show how dialogical values can be convincingly integrated with the human imperatives of purpose and instrumentality’ (p 17).
20 Quoted with the permission of Professor Peter Martin, Cornell University Law School.
21 S Van Rooyen, F Godlee, S Evans, N Black and R Smith ‘Effects of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial' British Medical Journal Vol 318, pp 23–27, 1999. Much of the literature stems from the medical fields, and the findings are mixed. See R A McNutt, A T Evans, R H Fletcher and S W Fletcher ‘The effects of blinding on the quality of peer review. A randomised trial’ Journal of the American Medical Association Vol 263, No 10, pp 1371–76, 1990, where it was found that blinding improves the quality of reviews.
22 The author concerned was in the process of moving post from one institution to another, and had problems accessing the journal's site because of altered authentication processes.
23 G Salmon E-moderating: The Key to Teaching and Learning Online Kogan Page, London, 2000.
24 G Salmon E-tivities: The Key to Active Online Learning 2nd edn, Kogan Page, London, 2004. See also J Pavey and S W Garland ‘The integration and implementation of “e-tivities” to enhance students’ interaction and learning' Innovations in Education and Teaching International Vol 41, No 3, pp 305–16, 2004.
25 See H Lefebvre The Production of Space Blackwell, Oxford, 1991; P Bourdieu ‘Social space and symbolic power’ Sociological Theory Vol 7, No 1, 14–25, 1989.
26 C Crook and P Light ‘Virtual society and the cultural practice of study’, in S Woolgar (ed), Virtual Society? Technology, Cyperbole, Reality Oxford University Press, Oxford, 2002, pp 153–75, 156.
27 Those invited by journal editors to act as reviewers may be interested to note that it is not uncommon for the articles sent to them for review to have already had an editorial eye cast over them.
28 This position is promoted by groups such as the Scholarly Publishing and Academic Resources Coalition, who work to correct what they describe as ‘market dysfunctions in the scholarly publishing system’. See their website: http://www.arl.org/sparc/.
29 The position is quite complex. Early research showed online journals did not have much impact, eg S Harter ‘Scholarly communication and electronic journals: an impact study’ Journal of the American Society for Information Science Vol 49, No 6, pp 507–16, 1997. There is research that shows attitudes regarding paper journals/online journals is changing; eg C McKnight and S Price ‘A survey of author attitudes and skills in relation to article publishing in paper and electronic journals’ Journal of Documentation Vol 55, No 5, pp 556–76, 1999. As McKnight and Price comment, ‘the results of [their] questionnaire suggest a small but increasing willingness to submit articles to electronic journals, but also suggest continuing concern about the permanence of such media. Almost a third of the sample felt that the addition of multimedia to their articles would be beneficial but few had the necessary skills to produce and incorporate multimedia objects. [They] concluded that authors should be involved more in future research and debate in electronic serial publishing’. Others have come to broadly the same conclusions. For example, Eason, Richardson and Yu categorised users into eight categories (the searcher, the enthusiastic user, the focused regular user, the specialised occasional user, the restricted user, the lost user, the exploratory user and the tourist). [K Eason, S Richardson and L Yu ‘Patterns of use of electronic journals’ Journal of Documentation Vol 56, No 5, pp 477–504, 2000.] Their research showed that the contents (both coverage and relevance) and ease of use of a system as they were perceived by the user were the most significant factors affecting patterns of use. Users' perceptions of both factors were affected by a range of intervening factors such as discipline, status, habitual approach towards information management, availability of alternative electronic journal services, purpose of use, etc. Their paper demonstrated the need for a service to meet the requirements of users with these varied patterns. Lawrence points out that there is on average 336% more citations to online articles compared to offline articles published in the same venue [S Lawrence ‘Free online availability substantially increases a paper's impact’, Nature, Vol 411, No 6837, p 521, 2001. See http://www.neci.nec.com/∼lawrence/papers/online-nature01/.
30 D Hunter, ‘Walled gardens’, Washington and Lee Law Review, Vol 62, pp 607–642, 2005.
31 Ibid at p 20.
32 JISC Disciplinary Differences Report, http://www.jisc.ac.uk/index.cfm?name=schol_comms_reports, Point 24.
33 Ibid, Point 32.
34 Ibid, Points 42 and 43
35 C Howell-Richardson and H Mellar ‘A methodology for the analysis of patterns of participation within computer mediated communication courses’ Instructional Science Vol 24, pp 47–69, 1996.
36 See for example F Becker and F Steele Workplace by Design: Mapping The High Performance Workspace Jossey-Bass, San Francisco, 1995. The literature on ‘situated learning’ also emphasises the effect of physical and social contexts. For them, learning is more likely to be deep and effective when situated in discipline-specific and authentic tasks. See J S Brown ‘Growing up digital: how the web changes work, education, and the ways people learn’ Change March/April, pp 11–20, 2000; J S Brown, A Collins and P Duguid ‘Situational cognition and the culture of learning’ Educational Researcher Vol 18, No 1, pp 32–42, 1989; S A Barab, K E Hay and T M Duffy ‘Grounded constructions and how technology can help’ TECHTRENDS March, pp 15–23, 1998. For a definition and discussion of authentic activities, see J Herrington, R Oliver and T C Reeves ‘Patterns of engagement in authentic online learning environments’ ASCILITE Conference, ‘Winds of change in the sea of learning: charting the course of digital education’ Auckland, New Zealand, 8–11 December, 2002.
37 See R W Rohfeld and R Hiemstra ‘Moderating discussions in the electronic classroom’, in Z L Berge and M P Collins (eds) Computer-mediated Communication and the On-line Classroom in Distance Education Hampton Press, Creskill, NJ, 1995; and M Hughes and N Daykin ‘Towards constructivism: investigating students’ perceptions and learning as a result of using an online environment' Innovations in Education and Teaching International Vol 39, pp 217–24, 2002.
38 M De Laat and V Lally ‘It's not so easy: researching the complexity of emergent participant roles and awareness in asynchronous networked learning discussions’ Journal of Computer Assisted Learning Vol 20, pp 165-71, 171, 2004. Some of the literature on student discussion forums is useful here. See W R Klemm ‘Eight ways to get students more engaged in online conferences’ The Higher Education Journal Vol 26, No 1, pp 62–64, 2001, also at http://academics.css.edu/PITL/resources/Eight_Ways.htm; N Prammanee ‘Understanding participation in online courses: a case study of perceptions of online interaction’ ITFORUM discussion paper, http://it.coe.uga.edu/itforum/paper68/paper68.html. See also Garrison, Anderson and Archer's model of community inquiry, which (following Dewey's practical inquiry model) divides community-based learning into three overlapping areas: social presence, cognitive presence, and teacher presence. See D R Garrison, T Anderson and W Archer ‘Critical thinking, cognitive presence, and computer conferencing in distance education’ American Journal of Distance Education Vol 15, No 1, pp 7–23, 2001, citing J Dewey How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process D. C. Heath, Boston, 1933.
39 Harnad, op cit, note 1.