THE NSDL EVALUATION PAGES
« Introduction » « NSDL Reports » « NSDL Consent Forms » « Online Evaluation Resources »

UCAR-approved Research Consent Forms

Surveys HSC-approved informed consent form for studies involving surveys
Interviews HSC-approved informed consent form for studies involving interviews
Observations HSC-approved informed consent form for studies involving observation

All research carried out under the auspices of UCAR that involves contact with human subjects requires approval from UCAR's Human Subjects Committee (HSC). This page details the application made by NSDL to the UCAR HSC in order to carry out the evalution work, and the HSC-approved informed consent forms given to research subjects in different research contexts.

The following protocol was submitted to UCAR's Human Subjects Commission in September 2005 and approved, after revisions, in November 2005.

Title: "The NSDL Evaluation Project"

1 Purpose and Significance of Project

The National Science Digital Library (NSDL: http://www.nsdl.org/) was created by the National Science Foundation to provide organized access to high quality resources and tools that support innovations in teaching and learning at all levels of science, technology, engineering, and mathematics education. It is a digital library of exemplary resource collections and services, organized in support of science education at all levels. Starting with a partnership of NSDL-funded projects, NSDL is emerging as a center of innovation in digital libraries as applied to education, and a community center for groups focused on digital-library-enabled science education.

According to the original NSDL project proposal:

To measure NSDL's progress towards these goals, the evaluation will focus on

The evaluation will review NSDL's resources, collections, and infrastructure; inventory its organizational capabilities and skills; evaluate its performance as a pathway to high-quality educational resources and collections; and produce recommendations for the next round of NSDL development.

This research will involve the evaluation of all aspects of NSDL's activities. The settings for the research will include all NSDL funded NSDL projects, including the NSDL educational collections of online resources for all educational levels, from kindergarten to graduate, and also the underlying architectures and services that support the search, discovery, and delivery of these collections. The library development teams include scientists, software programmers, administrators, librarians, teachers, and other specialists, who meet and interact in a variety of face-to-face and electronic meeting spaces.

The research will utilize a range of methodologies, including online surveys, interviews, usability studies, observations, paper-and-pencil surveys, and focus groups, that will investigate, analyze, and report on all stages of NSDL's activities.

The objectives of this research are to generate a project report for submission to NSF at the end of 2006, and also to develop a framework for identifying development opportunities with regard to the 'next generation' of NSDL that will help emphasize NSDL's role within a wider cyberinfrastructure context.

2 Methodology of the Project

2a General description of the structure of the project

This protocol only applies to evaluation research carried out by NSDL Core Integration. It does not apply to research carried out by individual NSDL projects, unless this research is carried under the direct supervision of NSDL Core Integration. In all cases, individual NSDL projects will obtain approval from their respective IRBs when carrying out their own research.

The evaluation will address five main areas of NSDL's activities. The first four areas deal with the process of digital educational resource creation, collection and use:

The final area deals with NSDL project activities aimed at developing, co-ordinating and reinforcing the knowledge and communication of NSDL as an organization, such as the core integration project componets.

The evaluation process will be guided by a conceptual model, the 'resource lifecycle model.' A detailed explanation of this model is available on the NSDL evaluation home page, http://eval.comm.nsdl.org/whitepaper.html. A central assumption of this model is that the generic task of any NSDL project should be to support the creation of valuable educational resources and infrastructure. The process of supporting and creating value could include the creation, review, and cataloging of resources that are exemplary in educational content, pedagogical effectiveness, scientific accuracy, technical functionality, and metadata. It could also include the provision of community and organizational services for NSDL that enable library resources to be used productively in educational context.

The resource lifecycle model focuses on the concept of the digital resource, tracking it from the moment of creation through to the moment of use, and then beyond to redesign and improvement. All stages of the resource lifecycle support NSDL's activities in one way or another. Individual NSDL projects may not necessarily address all the stages of the resource lifecycle, but the stages of the model provide useful conceptual boundaries for the implementation of evaluation activities, and also provide an integrative overview of how different NSDL activities may be evaluated as a coherent whole.

2b Description of the subject population, including recruitment methods, age, type, and number of subjects

All subjects will be adults. None of the participants will be minors. The subject participants will include:

Volunteer subjects will be contacted via an existing contact list of NSDL project PIs, as well as through NSDL community e-mail lists and other NSDL community forums (e.g. workshops, annual meetings, conferences, etc.) Funds permitting, subjects may receive honoraria for participating in focus groups or usability testing. Current rates for honoraria are $50-$100, to compensate for approximately two hours of participation.

The number of subjects to be recruited is as follows.

2c Description of the procedures involving human subjects (including procedures which may be deceptive, embarrassing or discomforting to participants). Describe what the participant will encounter: When, where, and how long.

The evaluation will include a number of methods that will be used to gain an understanding of NSDL's current activities, and of the usability, utility, and effectiveness of the library's services. The methods will include

Surveys

Surveys will be conducted online using an online survey vendor, Survey Monkey (http://www.surveymonkey.com). The subject will take part in the survey by sitting at their computer and accessing the survey through a web browser. The first screen will advise volunteers of their rights as subjects, and subjects will be directed to read an informed consent form before beginning the survey. Subjects will have to indicate that they understand the conditions under which the questionnaire is being administered before proceeding. A negative response will terminate the survey.

Questions will be addressed to NSDL PIs, and will be concerned with eliciting outlines of PIs' project activities, and also their satisfaction with the support that they received from the wider NSDL community. Participation will be voluntary. Participants will be informed that all results from questionnaires will be rendered anonymous.

Access to survey results will be password protected.

See "Informed Consent: Survey"

Interviews

Interviews may be carried out with either NSDL PIs or NSDL users, to elicit their understandings and/or experiences of NSDL as a library and as an institution. Interviews will take place in either the participant's place of work or a research setting, or may be conducted over the telephone. Interviews may be recorded and transcribed. Recordings and transcripts will be kept secure by the researcher.

Interviewees will be presented with an informed consent form for interviews before the interview is conducted. Interviewees will be notified of their right to refuse to participate in an interview. They will also be notified of their right to terminate the interview at any time.

See: "Informed Consent: Interview"

Usability Testing

Web site evaluation and usability testing will involve the observation of users interacting with versions of library software and hardware. Participants may be asked to 'think-aloud' and verbalize their thoughts relative to the tasks they are performing using the prototypes. User testing will take place principally in scheduled computer labs, or the usability laboratory at the DLESE Program Center, located at UCAR in Boulder, Colorado. The number of participants in this type of evaluation will typically be 10 or less, tested individually.

Potential participants will be briefed on the functionality of the software to be evaluated, the tasks they will be asked to perform, and the amount of time it will take. Participants may be asked to participate in one or more of the following: reading materials, answering questions, making decisions, indicating preferences, commenting about software, and completing learning tasks (e.g., drawing a diagram). We anticipate that participants will not find the experimental setting embarrassing or discomforting. Participants will be advised of their rights as subjects, emphasizing that: (1) it is the software that is being evaluated and not them, (2) there are no right and wrong answers, and (3) they can elect to stop the interview at any time, for any reason, with no questions asked.

Participants may be audio and/or video recorded. Recordings and transcripts will be kept secure by the researcher.

Access to all raw usability results will be confined to the contracted usability researcher and the project evaluator. Any reported usability results will be edited to preserve anonymity.

See: "Informed Consent: Observation"

Focus Groups

Focus groups will be conducted mainly with users, in order to gain understanding of the positive and negative ways experiences they encounter when using NSDL. During a typical focus group participants will be briefed on the purpose of the session, may engage in individual use of NSDL services, and then take part in a moderated group discussion. Focus groups typically will involve six to twelve participants. During the informed consent procedure, all participants will be verbally instructed that their participation is voluntary, that they may omit activities or questions for any reason, and that they may stop at any time for any reason.

Focus groups may be recorded and transcribed. Any recordings and transcripts will be kept secure by the researcher. Focus group participants will be presented with an informed consent form for observation before the interview is conducted.

See: "Informed Consent: Observation"

Ethnographic Observation

Ethnographic observation will be used to observe users using NSDL in 'real life' conditions, in order to understand the difficulties they encounter when they are constrained by time, computing resources, etc. Observations will be supported with note-taking, and perhaps also audio/video recording. Any recordings and transcripts will be kept secure by the researcher. Participants will be presented with an informed consent form for observation before the interview is conducted.

Observations of minors will not be conducted.

Ethnographic observation subjects will be presented with an informed consent form for observation before the interview is conducted.

See: "Informed Consent: Observation"

Use of Audio and/or Video Recording

Occasionally, audio and/or video recording will be used to record participants' verbalizations, non-verbal behavior, and interviews. Users will be informed in advance that audio and/or video tape recording will take place. Although past research has found that many participants are comfortable with being recorded, it is possible that some participants might find being audio and/or videotaped embarrassing. All users will be notified of their rights as participants, including their right to refuse to participate, or to participate without being recorded. Procedures for maintaining confidentiality of data, and, in the case of publication and/or the presentation of results in professional forums, will be explained; consent forms will be distributed; and refusal of consent from any person in the setting will lead to the cancellation of audio and/or video recording.

Observation will begin with the researcher introducing herself or himself, or being introduced by an NSDL member, as an ethnographer who is observing NSDL use. Procedures for maintaining confidentiality of the data, and, in the case of publication and/or the presentation of results in professional forums, will be explained. Refusal of consent by any one participant will lead to the cancellation of observation for that setting. The number of participants in the meetings will vary, between approximately 6 and 20 participants.

During ethnographic observation, participants will not be asked to engage in any additional procedures beyond participation in the institutional setting of which they are normally a part. Given that such settings constitute participants normal place of work, and that no new practices or procedures will be introduced by the observer, we do not foresee any procedures being deceptive, embarrassing or discomforting to participants. However, it is possible that some participants might find being audio and/or videotaped inherently embarrassing. As in all research, participants will be informed of their rights as participants. Informed consent will also emphasize procedures for maintaining confidentiality of data. Consent forms will be distributed; and refusal of consent from any person in the setting will lead to the cancellation of audio and/or video recording in that setting.

See: "Informed Consent: Observation"

Typical Lengths of Research Procedures

Webmetrics

Webmetrics are used to track the actions of users as they interact with a web site. Typical webmetrics may include: numbers of hits and visits to a site; paths followed by a user in a site; and operating system and browser configurations.

NSDL Core Integration is collecting webmetrics for visitors to nsdl.org, and to the sites of selected NSDL partners. These data provide useage statistics for project managers and project funders, and support the improvement of web site design. These data are always anonymous, and almost always aggregated. Two types of webmetrics are being collected: first-party web metrics, and third-party web metrics.

First party webmetrics

First-party webmetrics for nsdl.org are collected by NSDL Core Integration from servers directly under the control of NSDL Core Integration. Access to these webmetrics is password protected and restricted to NSDL Core Integration. These webmetrics data will only be made public (for instance as part of a report) in anonymous and aggredated form.

First-party webmetrics of individual NSDL projects are collected on their own project servers, and NSDL has no control over the webmetrics practices of individual NSDL projects.

Third-party webmetrics

NSDL Core Integration has contracted third-party contractor, Omniture (http://www.omniture.com/), to implement standardized webmetrics for nsdl.org and a selected number of NSDL partner projects. These webmetrics are implemented with a combination of javascript and session (non-persistent) cookies, that allows Omniture to track user interactions with nsdl.org site useage without having direct access to nsdl.org servers.

All data collected by Omniture belong to NSDL Core Integration, and not to Omniture. Access to Omniture webmetrics data is therefore only available to NSDL Core Integration, and is password protected. These data will only be made public (for instance in reports) in anonymous and aggredated form.

2d Description of any surveys, questionnaires or interview schedules to be used

Surveys

Representative questions include:

Usability questions

Representative usability questions include:

3 Description of the risks and benefits to participants

Risks

Subjects will be informed that one important risk arising from observation and/or audio and/or video recording, involves the loss of confidentiality of the subject(s). There are several aspects to this loss of confidentiality, ameliorations of which are discussed below. Participants will be informed of all steps the research will take to preserve confidentiality of data and to ensure anonymity in all published research.

Risk: Loss of confidentiality resulting from persons other than the researcher gaining access to research materials (field notes, audio and/or video tape recordings, completed interview schedules, etc.).

Amelioration: All research materials (field notes, audio and/or video tape recordings, completed interview schedules, etc.), will remain in the possession of the researcher. All such research materials will be stored in a locked filing cabinet in the researcher's office.

Risk: Loss of confidentiality resulting from inadvertent statements. Despite the subject(s) being informed of risks prior to observation, interview, and/or audio and/or video tape recording, subjects might make statements not intended to be distributed outside of the room. As subjects may be taking part in long meetings in which the researcher will often attempt to remain in the background, they may forget that they are being observed.

Amelioration: Additional attention will be paid in any published form of the data to ensure that all observations are anonymous and non-identifiable.

Risk: Loss of confidentiality resulting from presentation of data in professional forums (seminars, conferences, publications, etc.).

Amelioration: Additional attention will be paid in any published form of the data to ensure that all observations are anonymous and non-identifiable.

Benefits

As previously mentioned, where funds permit, and where not prohibited by the participant's institution, focus group and usability test participants will receive an honoraria (currently $50-$100) for participating. Otherwise there are no direct benefits for participating.

4 Description of means for ensuring privacy for subjects

Confidentiality

All materials obtained (from observation of meetings, interviews, etc.) will be used for research and analysis purposes and will not be made publicly available. All research materials (field notes, consent forms, audio and/or video tape recordings, completed interview schedules, etc.), will remain in the possession of the responsible researcher.

Storage of Confidential Materials

All confidential materials will be stored in such a way that they will be accessible only to the researcher. Such materials will be stored in:

Anonymity

In any published form of the data, attention will be paid during writing to ensure that all observations are rendered anonymous and non-identifiable.

5 Investigator's qualifications to conduct the study

This research will be coordinated primarily by Michael Khoo, Evaluation Co-ordinator for NSDL, under the supervision of Dr. Howe, Director of NSDL, and members of the research team. Others may be involved in the conduct of the research, and if so, will be trained in the methods by Khoo and will be supervised when carrying out the procedures.

6 Links to Consent Forms

Informed Consent Form for Surveys

Informed Consent Form for Ethnographic Observation

Informed Consent Form for Interview

Appendix: Sample Questions for Usability Testing

After the participant has worked through a learning module:

After the participant has used the NSDL tools:

Sample Questions for Interviews with NSDL Personnel