Learning from Other's Mistakes
- One Approach to teaching Information Literacy

Ewen MacDonald & Jarmo Saarti

Ewen MacDonald, Senior Assistant Department of Pharmacology & Toxicology, Kuopio University. P.O. Box 1627, 70211 Kuopio, Finland, Ewen.MacDonald@uku.fi

Jarmo Saarti, Library Director, Kuopio University Library, P.O. Box 1627, 70211 Kuopio, Finland, Jarmo.Saarti@uku.fi

Introduction

Everyone in the academic community is today confronted with an unprecedented problem. One click of the mouse can provide a student with enough source material to construct a doctoral thesis. Thus, the problem is no longer to locate the relevant material; today the difficulty is in separating the wheat from the chaff, or a more apt analogy, finding the few gold nuggets in the mountains of dross. The information literate student has to be able to evaluate web-based material which may be opinion disguised as fact. In this article we describe a short course entitled ‘Critical Journal Club’ and how after participating in this course, students become more critical, more sceptical and more information literate.

The number of journals has increased and the standard of refereeing is now more inconsistent than ever. In particular, one critical change has been the move from print to electronic formatting. This means that few journals employ sub-editors with a scientific background - instead articles are transmitted in pdf-format directly to the publisher. It is important that students are made aware that even material in eminent journals can occasionally contain substantial errors as well as numerous grammatical mistakes. We aim to stimulate discussion on the current level of peer reviewing as well as stressing the importance of integrating critical information literature skills into the curriculum.

Why is it so important that our students receive training in information literacy?

Information literacy is unlike many other academic disciplines, in that it is not a clearly structured entity - it cannot be taught in a single course. There are several definitions of information literacy but all include the following aspects: an awareness of how to recognize and define one’s own information needs; an appreciation that there are different types of documents and the ability to access them; the capability of using, in a critical manner, the found information and the competence to create new information, based on the previously published material. Thus Mackey and Ho state that both IL and IT skills are needed in the modern concept of web-literacy, i.e. web environment knowledge, web development knowledge, and research skills (Mackey & Ho, 2005).

Surprisingly, the needs of today’s students are not very different from their predecessors throughout the ages. Publication of research has always required a mixture of skills, critical reasoning and knowledge of what has already been done. What is new today is that web-based technology has greatly increased the amount of information available; the cultural situation (information society) has increased our economical dependence on information but the ideology of “all at once” has had an impact on the quality of the information published, especially on the quality of material on internet.

It is said that scientific journals can be differentiated from other journals by the so called peer-review process that each article undergoes. The idea of peer-reviewing is to emphasize the fact that science is a common process and scientific truths are achieved by the common efforts of the scientific community. As academics, we are expected to publish articles in these peer-reviewed journals at regular intervals, and more senior faculty members may be requested to join the editorial boards of these journals. Initially our students may find this time-honoured process of submission, review, revision, re-review and eventual acceptance of the paper after many weeks as a hopelessly outmoded concept. Today’s students are acquainted with modern technologies and electronic publishing - it seems to them rather old-fashioned to wait for months in order to publish one’s results when they can be disseminated around the world via internet within a few seconds.

However, while teaching our students about the advantages of peer reviewing and traditional formats, we should not instil in them an uncritical attitude to these journals. They have to be aware that simply because some material has been accepted for publication in an eminent journal, then its conclusions need not be unreservedly accepted as fact. This is an important facet of information literacy but it may be difficult for many students, as it is for many academics. Everyone is in a hurry and there is a tendency to read the abstract, ignoring the methodological and numerical details from which the conclusions are derived.

The credibility of good results can also be ruined if they are written in a slovenly manner. However, today it seems as if no one teaches students how to write good scientific English (or even good Finnish in the case of our own Finnish students). This basic art is not only being ignored, it is no longer valued. Teachers in the scientific disciplines erroneously assume that grammar and style are being taught in other courses. This may explain some of the problems we will mention later in this article, if students cannot be taught to recognize well written and accurate text, when they progress to reviewing the work of others, either as teachers of students or reviewers of articles, they will be unable or unwilling to demand corrections to substandard work.

This paper is intended to be provocative and should be read accordingly. It is not a scientific review of the accuracy of publications in the field of pharmacology. However, the scientific community has to be aroused from its beauty sleep and accept that changes, mostly detrimental, have occurred in the way scientific articles are refereed and published. Scientists have never been slow to grasp the advantages of new technologies and there is no reason to doubt that they will ignore the potential of open access publication in the future.

Critical journal club

In Finland, students of pharmacy spend five years in university, with the final year culminating in the writing of a pro gradu thesis and participation in and writing-up of a scientific research project. Our teaching of information literacy is targeted to converting the students into critical thinkers. This cannot be achieved in a single course, several courses are provided throughout their studies. In their first year, we provide an introductory course which lays the foundations, teaching how to use the available library resources. In their third year, when students have sufficient background knowledge, they can take a web-based course called ‘Pharmacology on the Internet’ which illustrates the pitfalls of relying on web-based material, stressing the ease and advantages of using PubMed as a source of reliable information (MacDonald & Saarti, 2003; 2005).

The final phase in our formal information literacy training is a brief course (40 study hours = 1.5 ECTS) called ‘Critical Journal Club’, offered in the spring term of the fourth year (i.e. just before they start their final year project works). Initially, this course was intended to acquaint students with interesting articles appearing in the major pharmacology journals. However, in recent years the ’critical’ aspect of the course has grown in importance, such that today its aim is no longer to review scientific breakthroughs but rather to turn students into enlightened readers of the literature. Each student is provided with an article published in the previous six months in an eminent pharmacological source journal, such as the European Journal of Pharmacology. There are three criteria involved in the selection of the article: 1) it must be available in the University's electronic collection so that it can be read by all students in the course, 2) the techniques utilized should be comprehensible to undergraduates and 3) it should not be very long, since timetable restrictions mean that we can only allocate thirty minutes to reviewing each article. No other selection criteria are applied; thus the articles to be assessed represent the vast majority of published pharmacology studies i.e. usually the effects of some form of drug treatment on cellular or in vivo responses in some animal model of disease. It needs to be stressed that no attempt is made to find articles likely to contain errors.

It comes as a major surprise to students that material in these respected journals can be erroneous - to quote one student who gave her presentation at the beginning of the course "though it may not seem like it from the figure, there is a significant difference between X and Y..." Closer inspection revealed that the student's eye had not deceived her; in fact there was an arithmetic error in the paper (Kitaichi et al., 2005). In a previous year, there was such a major error that after our intervention, the journal has promised to publish an extensive erratum, essentially nullifying the paper (Zhang et al., 2004). By the end of the course, all students have realized that it is a rare article indeed that does not contain some inconsistencies and some slip through the refereeing net still with fundamental errors. In this respect, the decline of refereeing standards can be turned into an advantage; students start to appreciate that all information needs to be scrutinized, evaluated and assessed and not accepted blindly simply because it is published in a peer-reviewed journal.

Once students become aware that articles in the literature can (and do) contain errors, they become adept at spotting these mistakes. Errors in this sense mean arithmetic, statistical or grammatical errors, not simply differences in the way the results can be interpreted. This is also a part of the course, a discussion of possible alternative conclusions. In general, this part of the session tends to be dominated by the teacher/mentor. Undergraduate students do not have sufficient background knowledge to construct alternative hypotheses to those of experienced researchers. Nonetheless, the fact that they are able to detect errors in the published papers illustrates to them in a very tangible way that all text has to be read, assessed and digested - nothing should be regarded as immune to criticism.

At the termination of this course, we surveyed the students (n = 11) to see which aspects of scientific papers they felt were crucial in their critical appraisal. Not surprisingly, problems which had been repeatedly encountered in the course were listed as reasons for scepticism i.e. failure to give any indication of biological variation or insufficient data precluding independent assessment of the results were major factors. If the results were only described in the text with no accompanying tables or figures, this was also a factor raising suspicion of authenticity (Fig. 1). The geographical origin of the paper had a slight effect, but this was not as great as the address of the authors (e.g. drug companies versus universities). Journals like Nature and Science were considered as reliable but beyond these high-impact journals, the students felt there was little difference in the reliability of material appearing in mid-impact or low-impact journals. Most students (81%) disagreed with the statement that the authors' own interpretation of the results is the best possible explanation. In summary, the course had taught the students that peer-reviewed journals are not gospel truth. They had learned that reading an abstract in PubMed is not sufficient if one wishes to critically assess the details of a scientific paper. Furthermore, if serious errors are noted in an article, it is our duty as informed members of the scientific community to alert the journal to these problems and expect that corrective actions will be taken.

From a pedagogic point of view, these are extremely useful lessons. The fact that the articles are chosen virtually at random but invariably at least one out of every ten contains such significant errors as to bring the conclusions of the entire paper into question emphasizes the concept of caveat emptor. It is unlikely if the students were confronted with specially selected examples of poor scholarship that it would have the same impact. To that extent, we can be grateful for the decline in the standard of refereeing which has occurred over the past ten years. In addition, apparently journals no longer employ skilled sub-editors, with papers being transmitted directly from the authors to the publishers. Thus, one of the papers for review this year actually contained a spelling mistake in its title (Tomonaga et al., 2005).

Traditional journals versus open access – open communication and peer-reviewing

It is rather surprising that while traditional journals have embraced certain aspects of electronic publishing such as automatic monitoring of manuscript review process, in other ways, they seem hidebound by tradition. Thus, we found it extremely difficult to alert the European Journal of Pharmacology, published by Elsevier, of the mistakes in published articles. The journal’s electronic home page does not permit readers to contact the journal; that privilege is reserved for authors and referees. Ultimately, the only effective means of contacting the journal was to send a letter to the departmental address of the editor-in-chief.

Equally, the journal appears to be in no hurry to publish the promised erratum. Though the authors have agreed that their article contained a horrendous error, almost a year has elapsed and there is still no indication that it will appear in any of the up-coming issues. Perhaps, this is not totally unexpected. Elsevier is a part of a commercial conglomerate whose principal aim is to make money for its shareholders. In fact, from the company’s standpoint, scientific journals can be viewed as a very good investment not primarily as valuable media for spreading truth and knowledge. The publishers can create a virtual monopoly and new technologies have been effectively introduced to allow them to publish their issues almost without any intermediaries and the labourers, i.e. the academics, perform most of the editorial work without any payment. According to investment analysts, journals in the fields of chemistry, physics and medicine have experienced the greatest price increases in the years from 1984 to 2002 with list prices rising by 6-fold and the belief is that this trend will continue in the foreseeable future (Morgan Stanley, 2002; Manna, 2003).

However, the ivory tower in which the traditional journals are ensconced may be about to crumble under the threat of open-access (OA) publishing. The basic doctrine of the OA movement has been to use the possibilities of internet in scientific publication to provide a platform to publish, disseminate and discuss scientific papers. Initially, the movement also emphasized the low cost of web-publishing: the access to the text is free for all, but quite often the scientist or his/her institution pays a fee that enables publication in the domain of the OA publisher. Lately the costs seem to be on the rise, which is not surprising: quality publishing means costs that have to be covered. There also has been a political ideology behind this evolution in OA publishing: to make it possible to disseminate information and to encourage comment, criticism and debate in a transparent atmosphere. When we compare this concept to the ideologies behind scientific publication - free publication of one’s ideas and the possibility to discuss and criticize in public the work of others, it is not surprising that many scientists have grasped the opportunities offered by this development.

Open access (OA) publication has sparked a lively debate in the mass media (Wray, 2006). At the European level, also governments have endorsed OA publishing and issued policy statements on behalf of the open dissemination of publicly-funded research (European Commission, 2006). One very important tenet that has been emphasized and implemented by some OA publishers (e.g. BioMed Central) is the ethical aspect of scientific publication. They have opened the peer-review process and attached it to the published document so that all readers can examine the changes to the article. They also have required open declaration in the text about competing interests and the authors’ role in the creation of the texts. In addition, many of the journals welcome readers’ comments and critiques since this provides a rapid, post-publication evaluation of the papers. The ultimate goal is to build a system which will be self-organizing and self-correcting according to the basic principles of science.

OA is now making a real impact and attracting submissions of material which previously would have gone to the traditional journals. Many OA journals have gained quite impressive impact factor ratings; there is some evidence that the open availability indeed increases the citation of the articles published in OA journals (Eysenbach, 2006). Thus, in many ways, the OA movement challenges the traditional ways of scientific publication, though it remains to be seen whether it will become the principal route for dissemination of research findings or it will remain a complementary system. The high quality journal Nature has recently announced that it will offer authors the possibility for open access peer review and will encourage web debate on its commissioned articles, evidence that the traditional journals are willing to incorporate many of the OA innovations (Campbell, 2006).

Thus, it seems that more and more commercial publishers as well as scientific professional bodies publishing scientific journals are now adopting methods used previously solely in OA-publishing or even more advanced methods mentioned in the Web 2.0 discussion. The main questions to be answered here are: can OA-publishing cover its costs and be viable and are we seeing a new era of commercial scientific publication that is more open. There is now a debate on how best to proceed to cover the costs with resources being collected from a wide diversity of sources and sponsors: users, institutions, publishers, writers and advertisers.

Discussion

Today's students come from an on-line world. They are web-savvy and are aware that their chat-room friend may not be the twenty year old blonde she purports to be. However, they do treat the printed word (even in electronic format) with undue respect. It is important that students are taught that even articles published in eminent journals may contain errors; they cannot be taken at face value. This new situation means that all those involved in educating university students must emphasize and integrate the aims of information literacy into their own tuition. Thus one new goal for all higher education should be how to endow students with critical knowledge management skills during the four or five years they are on campus. This also means that one must emphasize the basic values behind scientific research and publication: the search for truth, honesty and humility. Science cannot be sacrificed to the commercial interests; it is an invaluable human resource.

Pharmacy as a profession is also undergoing a major sea change. Previously, pharmacists had a very passive role in the health care system, simply dispensing medicines according to directions written by the physician. Nowadays, in several countries, pharmacists can prescribe a wide range of medicine and it is becoming the norm in Europe that they can alter the items on the prescription to a cheaper alternative if they consider the drugs to have equivalent efficacies. Today's students have to be taught to be life-long learners, their profession is changing at a rapid pace and they need to keep up-to-date with the latest advances by monitoring the professional literature. We teach them that the claims of therapeutic advances published in the press need to be viewed with caution; at the very least the original article needs to be read before one can satisfy oneself about the authenticity of such claims.

Modern publication procedures using interactive digital publication and an electronic dissemination system offer new opportunities to the scientific community to regain the control of the publication process of scientific texts and to make it more open as well as more self-organizing and self-correcting. There are some indications that the commercial scientific publishers are faced with the same financial pressures as other sectors of the economy and have resorted to the same cost-cutting management procedures favoured by banks and industrial manufacturers. Based on our experience, it seems that the sub-editing and proofreading have been outsourced by some publishers to low-cost non-English speaking countries which may account for the shoddy standards in some, so-called quality journals.

Ultimately, we return to the Descartian conclusion: all must be doubted and all that one can trust is one’s own critical and inquiring mind. However, one must hope that the scientific community will maintain its own standards and procedures at a high quality level. While it is healthy to be sceptical, one can only carry this so far; we really should be able to trust the conclusions made by our fellow scientists. The transparency associated with OA publications is one way to guarantee this will also be the case in the future.

References

Campbell, Philip: "Nature’s peer review trial." Nature’s web-page, June 2006. (August 8, 2007.) http://www.nature.com/nature/peerreview/index.html

European Commission: Study on the economic and technical evolution of the scientific publications market in Europe - final report. Brussels : EC Directorate C, January 2006, p. 108. http://ec.europa.eu/research/science-society/pdf/scientific-publication-study_en.pdf

Eysenbach, Gunther: "Citation Advantage of Open Access Articles." PLoS Biology, 4(2006)5. http://biology.plosjournals.org/perlserv?request=get-document&doi=10.1371/journal.pbio.0040157

Kitaichi, Yuji, Takeshi Inoue, Takeshi Izumi, Shin Nakagawa, Akiko Kato and Tsukasa Koyama: "Subchronic milnacipran treatment increases basal extracellular noradrenaline concentrations in the medial prefrontal cortex of rats." European Journal of Pharmacology, 520(2005),37-42.

MacDonald, Ewen and Jarmo Saarti: "Pharmacology on the Internet – a Web-CT Course Teaching Information Literacy for Pharmacy Students in the University of Kuopio". Bioscience Education e-journal, 1(May 2003). http://bio.ltsn.ac.uk/journal/vol1/beej-1-8.htm

MacDonald, Ewen and Jarmo Saarti: "Evaluation of a web-based course teaching information literacy to third year pharmacy students in the University of Kuopio, Finland". Pharmacy Education, 5(2005),1-5.

Mackey, Thomas P. and Jinwon Ho: "Implementing a convergent model for information literacy: combining research and web literacy". Journal of Information Science, 31(2005)6, 541-555.

Manna, M. La: "The Economics of Publishing and the Publishing of Economics". Library Revie, 52(2003)1, 18-28.

Morgan Stanley Equity Research: "Scientific Publishing: Knowledge is Power". Media, September 30, 2002. http://www.econ.ucsb.edu/~tedb/Journals/morganstanley.pdf

Tomonaga, Shozo, Tetsuya Tachibana, Hirokazu Takahashi, Momoka Sato, Donald Michael Denbow and Mitsuhiro Furuse: "Nitric oxide involves in carnosine-induced hyperactivity in chicks". European Journal of Pharmacology, 524(2005), 84-88.

Wray, Richard: "Brussels delivers blow to Reed Elsevier." The Guardian’s web-page, April 19, 2006. http://business.guardian.co.uk/story/0,,1756426,00.html

Zhang, Yi, Weijun Wang and Jianjun Zhang: "Effects of novel anxiolytic 4-butyl-alpha-agarofuran on levels of monoamine neurotransmitters in rats". European Journal of Pharmacology 504(2004), 39-44.