EVALUATION RESOURCES
A collection of online resources related do digital library evaluation. The focus is on peer-reviewed and published online research, stand-alone reports and white papers, special journal issues, and the evaluation projects of various digital libraries and museums.
STARTING POINTS
Reeves, Thomas, Apedoe, Xornam & Hee Woo, Young (2003) Evaluating digital libraries: A user-friendly guide.
http://dlist.sir.arizona.edu/398/
EXISTING NSDL AND DLESE EVALUATION WORK
DLESE Evaluation Services Home Page
http://www.dlese.org/cms/evalservices/
NSDL DigiQUAL Initiative
http://www.arl.org/stats/newmeas/emetrics/nsdl.html
NSDL - Standing Commmittee on Educational Impact and Evaluation
http://eduimpact.comm.nsdl.org/
NSDL - Webmetrics Taskforce
http://webmetrics.comm.nsdl.org/cgi-bin/wiki.pl
Shreve, G., M. Zeng, and B. Subrahmanyam (2004) Metadata Elements in the NSDL Metadata Repository: Results of a Preliminary Quality Analysis: Subject, Type, Audience.
http://dlist.sir.arizona.edu/608/
Sumner, Tammy, Mary Marlino and Kirsten Butcher (2003) Developing a Strategy for Evaluating the Educational Impact of NSDL.
http://eduimpact.comm.nsdl.org/evalworkshop/WorkshopPrelimReport.doc
OTHER RESOURCES
Arms, W (2000) Digital Libraries (online version) (originally published by The MIT Press). Chapter 8 covers User Interfaces and Usability.
http://www.cs.cornell.edu/wya/DigLib/index.html
Atkins, D., et al. (2003) Revolutionizing Science and Engineering Through Cyberinfrastructure (The Atkins Report).
http://www.nsf.gov/cise/sci/reports/toc.jsp
Blandford, A., & G. Buchanan (2003) Usability of digital libraries: A source of creative tensions with technical developments. IEEE Technical Committee on Digital Libraries, Vol. 1, No. 1.
http://www.ieee-tcdl.org/Bulletin/current/blandford/blandford.html
Budhu, M., & A. Coleman (2002) The Design and Evaluation of Interactivities in a Digital Library. D-Lib Magazine, Vol. 8, No. 11 (November 2002).
http://www.dlib.org/dlib/november02/coleman/11coleman.html
Council on Library and Information Resources (CLIR) CLIR works to expand access to information, however recorded and preserved, as a public good.
http://www.clir.org/
First Monday, Vol. 5, No. 6 Report on Web Wise 2000 (see section on "Friendly Users: Involving Users in Digital Libraries").
http://www.firstmonday.org/issues/issue5_6/
Frechtling, J., and L. Sharp (1997) User-friendly handbook for mixed method evaluations. National Science Foundation Report NSF 97153.
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153
Getty Office of Institutional Research Carries out research into and publishes reports on webmetrics for the Getty web site, and evaluates of technology use at the Getty Museum.
http://getty.edu/about/institutional_research/
Harvard Family Research Project (HFRP) HFRP examines topics related to family involvement in education, evaluation, and continuous improvement.
http://www.gse.harvard.edu/hfrp/
Institute of Museum and Library Services (IMLS) (n.d.) Perspectives on outcome based evaluation for libraries and museums.
http://www.imls.gov/pubs/pdf/pubobe.pdf [pdf file]
The International Center for Learner-Managed Learning
http://www.lle.mdx.ac.uk/iclml/Catall.htm
Joachims, T., L. Granka, B. Pan, & G. Gay (2005) Accurately interpreting clickthrough data as implict feedback. Proceedings of the conference on research and development in information retrieval (SIGIR) 2005.
http://www.cs.cornell.edu/People/tj/publications/joachims_etal_05a.pdf [pdf file]
Journal of Digital Information, Vol. 2, No. 4 Special Issue: Interactivity in Digital Libraries
http://jodi.tamu.edu/?vol=2&iss=4
Journal of Digital Information, Vol. 5, No. 3 Special Issue: Digital Libraries and User Needs
http://jodi.tamu.edu/?vol=5&iss=3
Journal of Digital Information, Vol. 5, No. 4 Special Issue: Social Aspects of Digital Information in Perspective
http://jodi.tamu.edu/?vol=5&iss=4
Khoo, M., and D. Ribes Studying Digital Library Users in the Wild - JCDL 2005 Workshop report. D-Lib, Vol. 11, No. 7-8.
http://dlib.org/dlib/july05/khoo/07khoo.html
Lib-QUAL Research Papers
http://www.libqual.org/Publications/index.cfm
Marchionini, Gary (2002) Co-evolution of user and organizational interfaces: A longitudinal case study of WWW dissemination of national statistics. Journal of the American Society for Information Science (Vol 53 Number 14, p.1192-1209) (Best JASIST Paper Award)
http://ils.unc.edu/~march/jasist_BLS_submit.pdf
Marchionini, Gary (2000) Evaluating Digital Libraries: A Longitudinal and Multifaceted View. Preprint from Library Trends, Fall 2000 vol 49(2), p. 304-333.
http://ils.unc.edu/~march/perseus/lib-trends-final.pdf
Marchionini, Gary, Ros Scaife, & Greg Crane (2000) Final Evaluation Report on the Perseus Project Publication Model, 1997-2000.
http://ils.unc.edu/~march/perseus/final_report.pdf
Marchionini, Gary, C. Plaisant, & A. Komlodi (2003) The people in digital libraries: Multifaceted approaches to assessing needs and impact. Draft of chapter in A. Bishop, B. Buttenfield, & N. VanHouse (Eds.) Digital library use: Social practice in design and evaluation, MIT Press.
http://ils.unc.edu/~march/revision.pdf
MetaScholar The MetaScholar study of user quality metrics is a collaborative venture of Emory University's MetaScholar Initiative and Virginia Tech, supported by the Institute of Museum and Library Services (IMLS), to determine what criteria underlie the preferences and assumptions of different groups of users regarding metasearch systems.
http://metascholar.org/quality_metrics/index.html
MCDU The MARC Content Designation Utilization Inquiry and Analysis Project. Empirical investigations of catalogers' use of the MARC bibliographic format's content designation. Includes frequency analyses of MARC record field utilization in over 50 million MARC records of Library of Congress and OCLC member libraries.
http://www.mcdu.unt.edu/?p=30
National Science Foundation (2004) Workshop on the Scientific Foundations of Qualitative Research.
http://www.nsf.gov/pubs/2004/nsf04219/
NetDay (2004) Speak-Up Day for Teachers. Insights and Ideas of Teachers on Technology: NetDay National Report on Speak Up Day for Teachers, 2004.
http://www.netday.org/sud4teachers_2004_report.htm
Nielsen, J (2005) The power of defaults. Jakob Nielsen's Alertbox, September 26, 2005.
http://www.useit.com/alertbox/defaults.html
Newman, M., K. Ambrose, T. Corner, J. Evans, P. Morris-Vincent, S. Quinn, J. Stephenson, & L. Vernon (2003) Evaluating educational impact: The approach followed by the Project on the Effectiveness of Problem-Based Learning (PEPBL). Paper presented at AERA 03, Chicago.
http://www.lle.mdx.ac.uk/iclml/mnaera03.pdf
Oliver, M., G. Conole, & L. Bonetti (n.d.) The hidden costs of change: Evaluating the impact of moving to on-line delivery.
http://www.shu.ac.uk/flish/oliverp.htm
Poll, Roswitha (2005) Bibliography: Impact and outcome of libraries.
http://www.ulb.uni-muenster.de/projekte/outcome/downloads/bibliography-impact+outcome.pdf
Sun, J., M. Heath, E. Byrom, J. Phlegar, and K. V. Dimmock (2000) Planning into practice: Resources for planning, implementing, and integrating instructional technology. SouthEast Regional Initiatives Technology in Education Consortium (SEIR-TEC)
http://www.seirtec.org/P2P.html
Stephenson, John, & J. Saxton (2004) An initial evaluation of participant experiences of UFL/LearnDirect's online learning through work programme leading to full university qualifications.
http://www.lle.mdx.ac.uk/iclml/ufi-ltw.pdf
