Printing Techniques for Formative Evaluation


Techniques for Formative Evaluation



Moderator Tamara Sumner, University of Colorado at Boulder

Sarah Giersch

Mary Marlino, University Corporation for Atmospheric Research

Carol Terrizzi, Cornell University

Manuel A. Pérez-Quiñones, Virginia Polytechnic Institute and State University

Ying Zhang, Rutgers University


User-Centered Design (UCD) is a methodology that focuses on user needs and goals when designing a system. Using UCD techniques for formative evaluation can help to assure that the system allows users to achieve their goals. Formative evaluation techniques assess the effectiveness of websites, resources, and services. Additionally, user interface design techniques and evaluative methods are important considerations for formative evaluation. Examples of Formative Evaluation techniques are cognitive walkthroughs, formal user testing with think-aloud protocol, and the Tullis method. Process evaluation will also be discussed with respect to the methodologies used and their effectiveness.

Notes - Techniques for Formative Evaluation


2003 Annual NSDL PI Survey, A Collaborative Formative Evaluation

Sarah Giersch

Details will be covered at Evaluation Meeting tomorrow

Question � Ellen

Were recommendations brainstormed or did they come out of open-ended comments?


Related Documents

Developing a Strategy for Evaluating the Educational Impact of NSDL

Mary Marlino

Participants stressed a necessary differentiation between Actual NSDL and Intended NSDL

Question

How do you integrate the CIL?


Information

Science + Art: Design and evaluation across disciplines

Carol Terrizzi

Idea that technical systems are just missing something that connects us to our humanity in the use of them

Question � Kaye

Has anything come out of how children think about these technologies?


Question � Tammy

Greg Crane advised us to think outside the box, Susan Jesuroga advised us to think beyond just a website

Any idea on how we infuse creativity in our own processes?

Comment � Ann

Literature on children looking at seductive details � the work has been done, not related to DLs, but it has been done, and we don�t have to reinvent the wheel

Question � Flora

Fascinating idea of using interpretive methods for evaluation, but how do we do it?

Comment � Mary

Makes me think how we can become more creative as an NSDL community � we have this opportunity to be creative in ways that few groups do

Usability Engineering: Formative Evaluation of NSDL Projects

Manuel A. Pérez-Quiñones

If you can�t measure it, you can�t make it better � because how do you know what better is

In the long run, usability methods reduce cost by knowing how to make something better

Findings from a study on iLumina

common problems were present in all the digital libraries tested

Question � Tammy

When you said reconciling developer and user mental models was easy to correct, I was surprised

Question

Chart you showed indicated that users don�t enter the process early, so what do we do to get the user mental model involved earlier?

Related Documents

Interlude


Tammy � I was intrigued by Carol�s picture of America�s most wanted painting, and do you really want that on your wall? We need to consider the implications of user input

Comment � danger in being everything to everybody � turns out badly

MIC Metadata Evaluation

Ying Zhang

MIC evaluation approach can be viewed as exploratory endeavor analyzing the verbal framework

Discussion


What are your needs regarding Formative Evaluation?

Liz � is it possible to do a mass critiquing? Have a session where we do a tutorial of evaluations based on projects that people bring

Ann � specification of research questions so that we can have the correlation of the research questions and the methods used

Susan � from experience, we knew we needed to do evaluation and assess user needs, but in many cases we don�t have the explicit questions thought out � so it would be good to work on the formulation of the questions

Susan � what methods are people using to provide incentive for feedback?

Liz � we�ve been doing web-based surveys � it turns out we have hundreds of people willing to evaluate

Manuel � a mailing list of people you want � a large mailing list will provide a reasonable response

Comment � we�ve had tough time getting teacher feedback

Comment � How has this impacted what goes on in classrooms? How has this impacted students? I would love to analyze it, but it�s a scary question.

Manuel � Before building anything, you need to find out what the problems you�re addressing are. I�m getting the feeling that is not how everyone here is working. If you have that, you will know your research questions. Going to the classroom and identifying the problem is key.

Comment � Learning Impact Studies take time and money

Flora � It is critical that we define our research and evaluation questions. Like Manuel, I�m not sure that that is our primary question, impact. What are we designing for? To get same old stuff to same old students. To get innovative stuff to same students. Or to get new students. We also need to spend time educating people about what our business is.

Comment � For my project, I need to define impact for me.

Tammy � This feedback from projects need to be considered by all projects and NSF

Ying � Impact includes instant impact (e.g. students can answer teachers questions on a subject) and longitudinal impact, which may be seen three years or 10 years later. We need to be careful when we make any conclusion for the educational impacts of a digital library.

Bing �

Comment � Educational impact has different levels. Students are a long-term thing. But there are also other considerations that interfere with the evaluation. Easier to measure in short-term is assessing whether we are making digital resources available to teachers � to broaden their field of knowledge, or for them to provide to their students. Three layers � have we made these things successful to the people who need them and do they find value in them? Have the teachers integrated them into their classrooms? Have the libraries and resources made an impact?

Ying � check Review of Educational Research � literature review

Ying - recommended resource - Aleven V., Stahl E., Schworm, S. et al. (2003). Help seeking and help design in interactive learning environments. Review of Educational Research, 73(3): 277-320

Comments

Please enter any comments in the following format.