Quality changes

Farewell

The Operational Research Society has a rule that no one can be an editor of one of its journals for more than 10 years. This is my 12th year as an editor, and I have been found out! So, this is my farewell editorial. It is tempting to thank all the people I have worked with on the journal, but the list would be excessively long and no one but those mentioned are likely to read it. And I could inadvertently forget someone ….

But my co-editors, who have one way or the other covered for me over the last 12 years, are Bob O’Keefe, Richard Baskerville, Hans van der Heijden, and Frantz Rowe and these are all people I would be happy to live on a desert island with. Which I am sure is more than they would be willing to say about me! If you ever get a chance to work with any of them, take it.

I have looked at the progress of the European Journal of Information Systems over the 21 years of its existence (20 volumes because volume 1 was spread over 2 years). Table 1 summarises the data.

Table 1 Basic data for the European Journal of Information Systems

Volume	Pages	Issues	Papers	Average pages per paper
1	426	6	35	11.77
2	314	4	26	11.23
3	302	4	25	11.48
4	247	4	20	11.80
5	285	4	23	12.13
6	270	4	23	11.61
7	274	4	24	11.17
8	303	4	24	12.38
9	260	4	20	12.00
10	236	4	18	12.44
11	298	4	21	13.67
12	324	4	24	12.42
13	318	4	22	13.73
14	525	5^a	59	8.51
15	664	6	53	11.43
16	833	6	58	13.78
17	698	6	48	14.17
18	614	6	39	14.92
19	706	6	42	15.64
20	489	6^b	29	16.07

^aThe Ciborra commemorative issue.

^bThe data are for 4 out of the 6 issues.

The journal appears to be unchanged for the first 13 years, with 4 issues per annum in which were published 18–25 papers a year in 236–324 pages. The average paper length appears to be slightly rising in the last few years, although the journal format changed in this period too. For the last 6 years the European Journal of Information Systems has appeared in 6 issues per annum, in which were published 39–58 papers a year in 614–833 pages. So with an increase of 50% in issues, the number of papers published has more or less doubled, and the number of pages printed has risen by roughly two and a half times. Another way of seeing this is that the average paper length has risen by about a third in the last 6 years. One explanation might be that as the refereeing process has improved (we have an Editor-in-Chief, Editors, Senior Associate Editors, Associate Editors and a Managing Editor in 2011, as opposed to two editors and an Editorial Board in 1991) papers are made to explain themselves better, and are therefore longer. But I like simple explanations, and this might be found when looking at the titles of the papers published in a 1991 issue (Table 2) compared with the titles of the papers published in a 2011 issue (Table 3). The latter have a different level of understanding of the subject to 20 years ago, so it is hardly surprising that the papers are longer. But there is no answer to this question, merely a personal choice as to which explanation or some other is preferred.

Table 2 Papers in Vol. 1 No 3 1991

An expert system for the U.K. Monopolies and Mergers Commission: a feasibility study

Success of software packages in small businesses: an exploratory study

Post-industrial and post-Fordist perspectives on information systems

MERISE: A European methodology for developing information systems

A review of object orientation and knowledge processing in office models

A review of information systems evaluation: content, context and process

Table 3 Papers in Vol. 20 No 4 2011

Focus groups and critical social IS research: how the choice of method can promote emancipation of respondents and researchers

An historically grounded critical analysis of research articles in IS

GUEST EDITORIAL

Special issue on the Kleinian approach to information system research – foreword

SPECIAL SECTION ARTICLES

The scholarly influence of Heinz Klein: ideational and social measures of his impact on IS research and IS scholars

Doing critical information systems research – arguments for a critical research methodology

Is information systems a discipline? Foucauldian and Toulminian insights

Enabling a Kleinian integration of interpretivist and socio-critical IS research: the contribution of Dooyeweerd's philosophy

Quality changes

Many things in life are a matter of personal taste or choice. This brings me to a discussion about the issue of research quality for the last time, at least as an Editor of this journal. When it comes to measuring the quality of a research paper, we should consider the following:

1. We have no criteria for evaluating the quality of a paper other than peer review. We could consider:

   • 1.1. Contribution to knowledge: how and who is to judge this (peer review – see 2 below) and over what time span (see 4 below)? We can sometimes know that a paper is making a contribution or not, but for most papers it is often hard to tell at the time (see 4 below).

   • 1.2. Rigour? It could be rigorously wrong.

   • 1.3. Relevance to the real world. It is difficult to imagine that much that is published could be accused of this.

   • 1.4. Well-written. It would be a good idea. Well written for whom, other researchers using the same priestly language, or whom?

2. Peer review is notoriously conservative. If the research does not fit current thinking, it is likely to be rejected.

3. Research progress comes from disestablishing the accepted beliefs of the research community, not by merely trying to show that the accepted beliefs are in some sense ‘correct’. However, if accepted beliefs are challenged, then peer review has a stake in rejecting the work, especially if it makes the reviewer's research appear to be wrong.

4. The impact of any research result has a variable life span, sometimes immediate, sometimes with a very long fuse. So a paper could easily have different quality ‘levels’ at different points in time.

5. What is believed to be good or bad quality is a dependency of too many variables: state of the world, fashionable research directions, reputation of the researcher, what people want to believe, the patience of the reviewer in working out what a paper is trying to say and so on.

6. If a paper goes through a number of review cycles, should not the reviewers become co-authors? Either the review process is adding something to the paper (so reviewers should be co-authors) or it is not (so why bother?).

7. A research paper is a stylised version of the work it describes (which makes it easier to read for those familiar with this writing style). But in the age of www this is misleading as well as not true. Search is the key web activity, and it is getting increasingly sophisticated. Maybe papers should give the search criteria and the research engines used.

One of the reasons for this quality conundrum is that paper quality gets confused with peer review. Obviously, quality journals review papers for publication and the method used is peer review by referees. But such peer review is not to measure the quality of papers, but whether the paper meets some (imaginary?) standard for that journal. This is not the same as measuring the paper's quality, but merely whether or not a threshold has been reached. I have sat on three panels/sub-panels for the U.K. Government's Research Assessment Exercise (RAE) in 2001 and 2008. The RAE is an attempt to measure the quality of research in Universities in order to allocate research resources. The RAE panels, and future such exercises have deliberately resisted using league tables since there is no confidence that they really tell you much. For example, a recent high-flying journal had many of its papers graded low because the journal had changed policy and wanted papers with research results but not research methods. No research methods, no research quality was the verdict.

Given these comments, you might wonder if research publications have any value at all. Our knowledge, tenuous as it is, is enhanced by an open debate among interested parties, from which one might expect a gradual improvement in understanding (cf On Giant's Shoulders). Such a debate is to my mind indivisible, and major contributions are I suggest less discernible in the general debating hubbub than some would like to assume.

To summarise, research publication is essential to research progress, but associating quality value (of a positive kind) can only be pretence.

Moving on, we have the vexed topic of league tables. There are league tables of universities, schools and so on, so why not journals? Why not indeed, journal league tables are just as useless. What is the point of reducing a multi-variable activity to one measure? Imagine applying such an approach to paintings, say. Which is better, the Mona Lisa or Van Gogh's Irises? What an absurd question, where would you start to try and reduce the question to a single measure of performance. If we follow current methods, we would end up with some weighted combination of the amount of paint used in the paintings (rigour), the number of visitors to see the paintings (citations), and the selected views of some art dealers (deans). League Tables are a level of banality that universities in particular should never accept. (But even on our web site we publish citation indices for this journal, as do most other websites!).

Valediction

What can I be proud of as a retiring editor? Three things I think:

1. A definition of Information Systems that I think is usable, that is ‘I.S. is I.T. in Use’ (Paul, 2010a, 2010b).

2. My list of things to remember when writing a paper (see Appendix below).

3. My willingness to always say what I believe is right, epitomised by the ‘Lack of Quality’ debate.

Appendix

How to write a paper that will be accepted (most likely)

So here is the latest version of Paul's ‘How do you write a quality paper?’

As I have mentioned before in the European Journal of Information Systems (Paul, 2009), you could do worse than making sure your paper would provide ‘obvious’ answers to the four questions below. In fact, very few published papers meet all four questions obviously. But they would be better papers if they did.

Assuming the paper is an appropriate paper for the journal being submitted to, then:

1. What story are you trying to tell the reader One story, note, not many. There may be two or three major points to the story, but much more than that confuses readers. A story written for the reader can be understood in 10 years time by the author if they need to revisit the paper. A story written for the writer (the majority published) will leave the authors as perplexed in 10 years time as readers are now.

2. What will the reader know after reading your story that they did not know before reading the story? The whole point of the paper one presumes.

3. Why should anyone believe you? This is the downfall of many papers, but if not believable, then that is the end of the paper.

4. Why should anyone care about the story being told? What value will the reader attach to the point of the story? If none, you will find that reviewers tend to disguise this problem under the rubric ‘significance of the contribution’.

5. What is the essence of your paper in one sentence. I know I said four questions, but the fifth question has nothing to do with whether your paper is published or not. Answering this question readily enables readers to attach the idea to you the author, and hence is likely to push your citation index up and enhance your reputation.

6. What motivated your research and the paper you have written? Again, I know I said four questions, but the sixth question may have nothing to do with whether your paper is published or not. Putting this information in your paper is likely to make the paper more interesting, and therefore it might be more widely read.