EIESC & TSC Webmetrics Telecon 1

Meeting Notes
Monday, March 08, 2004, 11.00 am-12.00 pm ET.

Notes taken by Laura Bartolo and Mike Wright

1. Summary

The EIESC and the Technology Standing Committees have been considering separately How are people using the library and What are the technology issues involved in determining usage? A joint task force between the EIESC and Technology SC is being established where EIESC provides guidance on types of information to be gathered and the Technology SC addresses the technology issues. Three major efforts were discussed and five broad areas were identified. A summary of the meeting is being posted on both SCs lists so that interested participants (including those that couldn’t participate in the call) can reply to the lists with the action items they would like to become involved with. A specific list for this work will follow later, but initial discussion on the current SC lists will allow for a broader initial participation.

2. Technology Standing Committee, Tools, Reusability - Discussion (Mike Wright, Ed Fox, Robby Robson)

The Technology SC is interested in determining what kind of information should be collected by a project for NSDL use logging and log analysis. This information will be useful to those projects that already have systems so they can determine how to instrument their systems for capturing log information, and for those projects projects building tools to be used by others (e.g. CWIS tool from Infoscout group).

There is also the concern of what tools are available, or need to be built, for the analysis of log data. Groups are using a variety of off-the-shelf tools with different levels of capabilities. While all the tools are capable of reporting similar basic metrics, they still require time and effort to use effectively to generate reliable reports, and they vary in their abilities to perform more complex analyses.

The Citadel project is investigating log standards so that comparisons across sites can be made and improvements can be documented. Tools have been developed to implement the data gathering within their projects. These methods and tools could be used as a basis for considering what we should do for NSDL. More information on their work can be found at Digital Library XML Logging Standard and Tools

The Reusability project is interested in documenting educational impact of the NSDL by developing tracking standards for content. Tracking results of content usage and time spent with resources would add greater understanding of the educational value of objects. There is the specific task of how learning systems (that already have some tracking and logging facilities) are part of the NSDL consideration of impact.

3. EIESC Pilot Log Study - Discussion (Tammy Sumner, Mimi Recker, Jim Dorward, Casey Jones)

A pilot log study was conducted with 6 reasonably mature digital library projects volunteering to take part for six months. The pilot collected basic information that was of interest to most DLs and was easy to collect. The instrument used in the pilot is available on the EIESC workspace.

The overall finding of the pilot was that it was difficult for projects to report simple data. None of the projects could report all of the data for all of the months. Groups had a difficult time setting things up (instrumenting their systems), and had difficulty in identifying an individual in their group with the time to do this. All of the projects used different tools. Only one project excluded internal or robot activity. In summary, DLs would like to know who uses their collections and how they compare with similar projects but there is no money, time or expertise to support the activity.

4. Participants interests – discussion (Susan Jesuroga, Martha Kyrillidou)

Martha pointed out that in ARL’s (Association of Research Libraries) experience, it takes time and iteration to develop data standards that can be compared reliably. She also described Project COUNTER initiative involving publishers and librarians in the UK to develop a framework and best practices to support reporting usage statistics in a consistent fashion

FOLLOW UP - more information about Project COUNTER
and more information about the ARL E-Metrics project. (Martha Kyrillidou)

About COUNTER

The use of online information resources is growing exponentially. It is widely agreed by producers and purchasers of information that the use of these resources should be measured in a more consistent way. Librarians want to understand better how the information they buy from a variety of sources is being used; publishers want to know how the information products they disseminate are being accessed. An essential requirement to meet these objectives is an agreed international Code of Practice governing the recording and exchange of online usage data. COUNTER has developed just such a Code of Practice.

Release 1 of the COUNTER Code of Practice, which focuses on journals and databases, was agreed in December 2002 and is published in full on this website. Future releases of the Code of Practice will extend its coverage to other categories of online content.

Brief update on the ARL E-metrics project

Susan said that in the long term CI would want a clear set of requirements and an understanding of the current use of tools by projects in order to gauge the scope of any undertaking by CI for any centralized analysis and to determine the personnel that would be needed if that became a need. The Pathways projects may be an appropriate set of groups to work out these practices.

5. Possible next steps

Discussions identified several areas where work could begin. The group will use email on the lists to move forward on a spectrum of data collection measures and to identify who would like to work in which areas. This will enable the group to begin work in the short term while refining the instruments for long term use:

1. Information gathering (literature review, best practices)
2. Understanding the scope of logging and analysis, what is needed, and why
3. Rerunning the pilot study with original 6 participants and a call for volunteers
4. Listing of projects developing tools e.g. CITADEL
5. Sampling log data (at specific times in year) from various participants to get an overall picture of NSDL

On the Call

Laura Bartolo
Elizabeth Blackmer
Jim Dorward
Ed Fox
Susan Jesuroga
Casey Jones
Liz Liddy
Martha Kyrillidou
Mimi Recker
Robby Robson
Tammy Sumner
Susan Van Gundy
Steve Weimar
Mike Wright

More Follow Up

(Anita Coleman)

Yahoo recently indexed all NSDL resources from the metadata repository and gave us access to The Site Match Center (SMC). This is a graphical reporting system for managing and tracking how many "clicks" NSDL resources receive because they are being discovered in Yahoo. It provides summary reports to effectively manage search marketing performance.

Since our resources were indexed in Yahoo on 2_28_04, NSDL resources have been "clicked on" 2,668,444 times. Users went to live urls 63,014 times. Average successful urls: 30,336. Average clicks per url: 2.9321.