|
Techniques for Formative EvaluationModerator Tamara Sumner, University of Colorado at Boulder Sarah Giersch Mary Marlino, University Corporation for Atmospheric Research Carol Terrizzi, Cornell University Manuel A. Pérez-Quiñones, Virginia Polytechnic Institute and State University Ying Zhang, Rutgers University User-Centered Design (UCD) is a methodology that focuses on user needs and goals when designing a system. Using UCD techniques for formative evaluation can help to assure that the system allows users to achieve their goals. Formative evaluation techniques assess the effectiveness of websites, resources, and services. Additionally, user interface design techniques and evaluative methods are important considerations for formative evaluation. Examples of Formative Evaluation techniques are cognitive walkthroughs, formal user testing with think-aloud protocol, and the Tullis method. Process evaluation will also be discussed with respect to the methodologies used and their effectiveness. Notes - Techniques for Formative Evaluation2003 Annual NSDL PI Survey, A Collaborative Formative EvaluationSarah GierschDetails will be covered at Evaluation Meeting tomorrow Question Ellen Were recommendations brainstormed or did they come out of open-ended comments?
Related Documents Developing a Strategy for Evaluating the Educational Impact of NSDLMary MarlinoParticipants stressed a necessary differentiation between Actual NSDL and Intended NSDL Question How do you integrate the CIL?
Information Science + Art: Design and evaluation across disciplinesCarol TerrizziIdea that technical systems are just missing something that connects us to our humanity in the use of them Question Kaye Has anything come out of how children think about these technologies?
Question Tammy Greg Crane advised us to think outside the box, Susan Jesuroga advised us to think beyond just a website Any idea on how we infuse creativity in our own processes? Comment Ann Literature on children looking at seductive details the work has been done, not related to DLs, but it has been done, and we dont have to reinvent the wheel Question Flora Fascinating idea of using interpretive methods for evaluation, but how do we do it?
Comment Mary Makes me think how we can become more creative as an NSDL community we have this opportunity to be creative in ways that few groups do Usability Engineering: Formative Evaluation of NSDL ProjectsManuel A. Pérez-QuiñonesIf you cant measure it, you cant make it better because how do you know what better is In the long run, usability methods reduce cost by knowing how to make something better Findings from a study on iLumina
common problems were present in all the digital libraries tested
Question Tammy When you said reconciling developer and user mental models was easy to correct, I was surprised
Question Chart you showed indicated that users dont enter the process early, so what do we do to get the user mental model involved earlier?
Related Documents
InterludeTammy I was intrigued by Carols picture of Americas most wanted painting, and do you really want that on your wall? We need to consider the implications of user input Comment danger in being everything to everybody turns out badly MIC Metadata EvaluationYing ZhangMIC evaluation approach can be viewed as exploratory endeavor analyzing the verbal framework
DiscussionWhat are your needs regarding Formative Evaluation? Liz is it possible to do a mass critiquing? Have a session where we do a tutorial of evaluations based on projects that people bring Ann specification of research questions so that we can have the correlation of the research questions and the methods used Susan from experience, we knew we needed to do evaluation and assess user needs, but in many cases we dont have the explicit questions thought out so it would be good to work on the formulation of the questions Susan what methods are people using to provide incentive for feedback? Liz weve been doing web-based surveys it turns out we have hundreds of people willing to evaluate Manuel a mailing list of people you want a large mailing list will provide a reasonable response Comment weve had tough time getting teacher feedback Comment How has this impacted what goes on in classrooms? How has this impacted students? I would love to analyze it, but its a scary question. Manuel Before building anything, you need to find out what the problems youre addressing are. Im getting the feeling that is not how everyone here is working. If you have that, you will know your research questions. Going to the classroom and identifying the problem is key. Comment Learning Impact Studies take time and money Flora It is critical that we define our research and evaluation questions. Like Manuel, Im not sure that that is our primary question, impact. What are we designing for? To get same old stuff to same old students. To get innovative stuff to same students. Or to get new students. We also need to spend time educating people about what our business is. Comment For my project, I need to define impact for me. Tammy This feedback from projects need to be considered by all projects and NSF Ying Impact includes instant impact (e.g. students can answer teachers questions on a subject) and longitudinal impact, which may be seen three years or 10 years later. We need to be careful when we make any conclusion for the educational impacts of a digital library. Bing Comment Educational impact has different levels. Students are a long-term thing. But there are also other considerations that interfere with the evaluation. Easier to measure in short-term is assessing whether we are making digital resources available to teachers to broaden their field of knowledge, or for them to provide to their students. Three layers have we made these things successful to the people who need them and do they find value in them? Have the teachers integrated them into their classrooms? Have the libraries and resources made an impact? Ying check Review of Educational Research literature review Ying - recommended resource - Aleven V., Stahl E., Schworm, S. et al. (2003). Help seeking and help design in interactive learning environments. Review of Educational Research, 73(3): 277-320 CommentsPlease enter any comments in the following format.
|
