Home    Basics Search Strategies EResources Things to Try Contact Linda Hansen Special Helps Evaluating Sources and Resources

Related document:
Critical Thinking

Content Evaluation

Introduction

Evaluation of materials should be based on objective criteria and conducted within a particular context. To ask whether a research resource is "good" is to ask a meaningless question -- unless we can determine how "good" is defined. Generally, evaluation of a research source or resource should be based on:

• Authority -- the reputation of the creator, author, compiler, publisher, etc.
• Audience -- the vocabulary, sophistication of the content, and subject matter should match the intended audience
• Currency, frequency, and standardization -- the material should be timely and topical, up dated when appropriate, and appear in a standard form thus making it more accessible to its audience
• Format and container -- for example, research material in book form generally should be presented with tables of contents, indexes, and the like
• Depth, breadth and length -- should meet the standards appropriate to the subject and presentation of the work
• Content value -- often based on the reader's purpose, subject knowledge, and critical abilities

The presence of an item an internet searcher, on a handout, as a footnote in a journal article, or as a link on an institutional site, does not constitute a recommendation nor a guarantee of accuracy and currency of content. Information on the internet, or in journals, is controlled by those who post, or publish, it. It may be wonderful and it may not.

When you evaluate a source or resource, you need to consider its purpose before deciding its worth: is it there to educate, inform, persuade, entertain, advertise or market? Does it have an agenda, either clearly expressed or hidden? Are you looking at a peer-reviewed or scholarly source, or a student paper produced for an undergraduate course? Has the author undertaken to present a balanced paper or has s/he ignored items which did not support her/his theory?

In broad terms, evaluation is determining the value of the information for your purpose. Citation is a necessity when using the information for any research or academic purpose. Generally, citation elements are extracted from evaluation elements. So ... does the material include all needed citation elements (author, title, date, place of publication/publisher)? If it does not, then maybe you should consider the value of what you are reading very carefully.

Logical Fallacies

Evaluating a research resource, electronic or otherwise, is more than simply evaluating the nature and worth of the publication, and the reputation of the publisher and the researcher. At some point, you have to decide whether the data presented were gathered appropriately and whether the conclusions drawn are logical, appropriate and actually a result of the data collected. Even the very best researchers may make mistakes, overlook the obvious, and ignore evidence. The result may be any number of logical fallacies which, in turn, may invalidate both the data and the conclusions.

Top of Page

Electronic Clues and Cues

Country codes may appear, sometimes in conjunction with major domains and subdomains, and sometimes not. If you are doing research that is nationally dependent (for example, law-related) or your linguistics skills are limited, it helps to pay particular attention to the origin of the site you are using. While not every web site registered in a particular country will contain information specific to that country, there may be a greater likelihood that it does. The country codes are listed in ISO 3166, or in Root-Zone Database from IANA. A few examples of country codes are given here:

Other standard codes and symbols may also appear, such as:

After you've checked the address, check out the site for:

• Appropriate material for the target audience
• Reputation of publisher/compiler
• An index
• Overall size of the site
• Frequency of updating/lag time
• Language appropriate to the target audience
• Comprehensiveness
• Overlap with similar sites/Unique
• Format
• Design and content appropriate to selected structure (hypertext/gopher/ftp)

Running a Web Site

Running a good web site takes an investment of time and money. While time and money do not guarantee quality, the investment made by an organization in its site will give you some indication of the esteem in which the site is held within the organization itself.

Look for statements of responsibility: an author, creator, webmaster, maintainer, with an address and a clickable link to allow comment and questions about the site from users.

Also consider such things as:

• Longevity of the site
• Archives policy
• Evidence of knowledge of the primary audience
• Information/Help available through email (and the response time)
• Site administrator address given (and response time where appropriate)
• Reasonable requests for user information (for example, the site does not ask questions about your job, your age and your income before allowing you to register for "free" use of its services)
• Privacy policy where appropriate
• "Warranty" on accuracy of information mounted
• Policy on changed/superceded links (pointers offered to new site/new address of old site)

Cost/Time Effectiveness

Using a web site requires an investment of time and often, money by the user. That being the case, part of any evaluation should be a consideration of how hard or easy the site is to utilize effectively. Consider such things as:

• Charges to use the web site (these should be upfront and unambiguous)
• Hardware/Software required to utilize resources
• Design appropriate for multiple browser types/versions
• Cost of other available resources

Top of Page

More Information

General

• Critical Thinking
• How to Evaluate Journal Articles, Colorado State University Libraries
• Evaluating resources, UC Berkeley Library
• Critically Analyzing Information Sources, Cornell University Library
• PLoS Collections: Ghost Writing, PLoS
• Selection Criteria for Quality Controlled Information Gateways, DESIRE - RE 1004, UKOLN Metadata Group
• Evaluating Sources, Australia National University Library
• Evaluating Information Found on the Internet, Sheridan Libraries, Johns Hopkins

Arts and Humanities

• Primary Sources on the Web: Finding, Evaluating, Using, Reference and User Service Association History Section, ALA
• History Standards National Standards for History, National Center for History in the Schools, University of California, Los Angeles

Legal Information

• Evaluating Websites and Blogs, University of Akron School of Law
• Gatekeepers of Legal Information: Evaluating and Integrating Free Internet Legal Resources into the Classroom, Lee Jootaek, School of Law Faculty Publications, Northeastern University, Barry Law Review
• Creating a Safety Net - A Proposed Rating Form for Assessing the Quality of Legal Information in Websites, Alan Robinson, Journal of Information Law and Technology, 2000 (1)
• International Legal Research Tutorial, Duke University School of Law and the University of California, Berkeley, School of Law

Medical Information

• Evaluating Health Information, MedlinePlus
• HON Code of Conduct for medical and health web sites, Health on the Net Foundation
• A User's Guide to Finding and Evaluating Health Information on the Web, Medical Library Association
• How to read a paper, a collection of articles from the British Medical Journal
• PLoS Collections: Evaluating eHealth, PLoS
• PLoS Collections: Disease Mongering, PLoS
• Behind the Headlines, PubMed Health
• Behind the Headlines, NHS Choices UK
• Sense About Science: Equipping people to make sense of science and evidence
• I Don't Know What to Believe: Making sense of science stories, Sense About Science, 19 April 2006
• An International Comparison of Web-based Reporting About Health Care Quality: Content Analysis, Olga C Damman, Ylva KA van den Hengel, A Jeanne M van Loon, Jany Rademakers, J Med Internet Res. 2010 Apr-Jun; 12(2): e8. doi: 10.2196/jmir.1191
• Belief beyond the evidence: using the proposed effect of breakfast on obesity to show 2 practices that distort scientific evidence, Andrew W Brown, Michelle M Bohan Brown, and David B Allison, First published September 4, 2013, doi: 10.3945/ajcn.113.064410 Am J Clin Nutr November 2013 ajcn.064410
• Second-hand smoke in cars: How did the "23 times more toxic" myth turn into fact?, Ross MacKenzie, Becky Freeman, CMAJ 2010 May 18; 182(8): 796–799. doi: 10.1503/cmaj.090993
• Typologies of Extreme Longevity Myths, Robert D. Young, Bertrand Desjardins, Kirsten McLaughlin, Michel Poulain, Thomas T. Perls, Curr Gerontol Geriatr Res.2010: 423087. doi: 10.1155/2010/423087
• A basic introduction to research: how not to do research, Andrew J. Vickers, J Soc Integr Oncol. 2008; 6(2): 82–85
• An Analysis of News Media Coverage of Complementary and Alternative Medicine, Billie Bonevski, Amanda Wilson, and David A. Henry, PLoS ONE, 2008; 3(6): e2406, doi: 10.1371/journal.pone.0002406
• Guidelines for Medical and Health Information Sites on the Internet: Principles Governing AMA Web Sites, Margaret A. Winker, JAMA 283, 12 (22-29 March 2000)
• Published criteria for evaluating health related web sites: review, Paul Kim, et al., British Medical Journal, 318 (6 March 1999):647-649
• Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information, Gunther Eysenbach, et al., British Medical Journal, 317 (28 November 1998):1496-1502
• Defending legitimate epidemiologic research: combating Lysenko pseudoscience, James E Enstrom, Epidemiol Perspect Innov. 2007; 4: 11, doi: 10.1186/1742-5573-4-11

Science, Engineering, and Technology

• Sense About Science
• Managing for High-Quality Science and Engineering at the NNSA National Security Laboratories, Committee to Review the Quality of the Management and of the Science and EngineeringResearch at the Department of Energy's National Security Laboratories -Phase 1; Laboratory Assessments Board (LAB); Division on Engineering and Physical Sciences (DEPS); National Research Council; National Academies, 2013
• The Future of Scientific Knowledge Discovery in Open Networked Environments: Summary of a Workshop, Paul F. Uhlir, Rapporteur; Board on Research Data and Information; Policy and Global Affairs; National Research Council, 2012
• On Being a Scientist: A Guide to Responsible Conduct in Research, Committee on Science, Engineering, and Public Policy, National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 3rd ed., NAP, 2009
• Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age, Committee on Ensuring the Utility and Integrity of Research Data in a Digital Age; National Academy of Sciences, 2009
• The Why Files: Science behind the news, University of Wisconsin
• Bad science in the headlines. Who takes responsibility when science is distorted in the mass media?, Andrew Moore, EMBO Rep. 2006 December; 7(12): 1193-1196, doi: 10.1038/sj.embor.7400862
• Committee for Skeptical Inquiry
• Annals of Improbable Research
• Science and Technology: Public Attitudes and Public Understanding: Science Fiction and Pseudoscience, Science and Engineering Indicators 2002, National Science Foundation
• Teaching About Evolution and the Nature of Science, National Academy of Sciences, 1998

Peer Review Process

• Peer Review,Office of Research Integrity, Department of Health and Human Services, US
• Peer Review Quick Guide: Detecting Common Mistakes and Considering Dilemmas, Department of Health and Human Services, US
• Center for Scientific Review, NIH
• Peer Review Policies and Practices, NIH
• Selected References on NIH Peer Review: The Administration of Research at NIH, 1947-1997, NIH
• Who's Afraid of Peer Review?, John Bohannon, Science Magazine, 4 October 2013, 342 (6154): 60-65
• The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal, Lutz Bornmann and Hans-Dieter Daniel, PLoS ONE, June 2010, 10.1371/journal.pone.0011344
• Signs of epistemic disruption: Transformations in the knowledge system of the academic journal, William W. Cope and Mary Kalantzis, First Monday, vol. 14, no. 4, 6 April 2009
• The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality, Michael L. Callaham and John Tercier, PLoS Medicine, January 2007
• Peer Review and the Acceptance of New Scientific Ideas, Discussion paper from a Working Party on equipping the public with an understanding of peer review, Sense About Science, 3 June 2006
• A Strategy for Assessing Science: Behavioral and Social Research on Aging, Committee on Assessing Behavioral and Social Science Research on Aging, Irwin Feller and Paul C. Stern, editors, National Research Council, 2006
• Strengthening Peer Review in Federal Agencies that Support Education Research, Lisa Towne, Jack M. Fletcher, and Lauress L. Wise, eds., Committee on Research and Education, National Research Council, 2004
• Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices, Committee on Research and Peer Review in EPA, Board on Environmental Studies in Toxicology, National Research Council, 2000
• Scholarly Publishing, Peer Review and the Internet, Peter Roberts, First Monday, vol.4, no. 4, 5 April 1999
• Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation, J. Scott Armstrong, Science and Engineering Ethics, 3 (1997)

Top of Page

This page copyright, created and maintained by Linda Hansen.
Comments and suggestions to: lhansen16@gmail.com
Created: 1997/04/11 Last updated: 2017/10/20
Terms of Copyright
This document: ...basics/eval.htm