Printing DL Requirements: Evaluating System Design and Use
DL Requirements: Evaluating System Design and Use
Marcia Mardis, Merit Network, University of Michigan
Bing Pan, Cornell University
User-Centered Design (UCD) is a methodology that focuses on user needs and goals when designing a system. User needs and goals affect the design of back-end system elements (hardware and software) as well as front-end elements such as the placement of information for accessibility and functionality. Evaluation methods used to analyze system requirements are similar to those analyzing the use of digital libraries because both are concerned with what users are doing or might do with the system.
Comments
Dr. Gay is unable to come to this session. Bing Pan will present her work.
Agenda for Session!!
- Marcia Mardis - Overview and Introduction
- Bing Pan
- Marcia Mardis
- Bing Pan
Notes - DL Requirements: Evaluating System Design and Use
Marcia Mardis
- librarian
- work on teachers' information seeking behaviors
- Michigan Teacher Network, Merit Network, UM
Bing Pan
- postdoc in HCI program at Cornell
- Ph.D. from University of Illinois at Urbana-Champaign
Building the Missing Link: A Conceptual framework for the Use of DLs and a Research Agenda
Bing Pan
Successful DLs
- such as amazon.com are useful and convenient
- phonebook is a most useful technology - ubiquitous and user-friendly - it is a successful system
- we can learn about user needs from this
NASA DL and INSPEC
- articles are indexable by search engines - interface becomes relevant
- replacement effect of information finding tools when better technology comes along
most users accessing NSDL.org are from edu domain
3 user groups are insiders, outliers, and outsiders
currenlty, few real users - this is not as intended
information foraging theory is used to explain how people look for information - via cues and clusters
- from this comes a conceptual framework for the types of information search that a DL is useful for - time reduction from use of physical library
- users don't know about digital libraries, they know Google
DL creators need to think about marketing their DLs - what they provide that is unique and of value to users
user studies are being used to study their conceptualization of information clusters
Study I - Profiling Information Environment - methods: interviews and surveys
Study II - Dynamic Mental Models and Information Cluster Choices
- methods include clickstreams, webcam, camcorder to capture user's information seeking behaivor
Design Implications
- finding niche market of the digital library
- marketing
- determinants of satisfactory information search experience
- customized interfaces
Question from Manuel
How specific do we build things?
When earlier today CI said we could generalize from items developed for MS, I don't see that in my research, each user group needs something different
How generic can you make an interface?
Transaction Log Analysis & Michigan Teacher Network
Marcia Mardis
"I run a DL for educators and we need to know what the heck they do with it"
UCD process needs to go along with development of the project
Transaction Log Analysis (TLA) when I talk about it refers to the search process - queries
3 main categories that you can get from TLA
- session characterizations
- query characterization
- search term characterization
Excite studies articulated frameworks and general taxonomies for TLA that other researchers look to
- 1997 study showed adult-related issues
- 1999 study showed business-related issues
New Zealand DL studies draw line for TLA looking at a broad search engine and looking at TLA for a DL
- they are 2 different things
- search of metadata v search of object
AltaVista studies show difference from predictions of user personality and behavior differs from actual user behavior
Artemis studies show teacher and student learning behavior
- do teachers really care about metadata? concluded yes
- what help do teachers need to get through search process?
consistent findings
- few used advanced search features
- when boolean used, it is used badly
(Marcia's findings found the same)
- need to capture users on first page
- when users don't find what they want in one query, they usually leave the site
- most queries are short
TLA Caveats
- do you have implied AND or OR that users might not guess at? you need to consider how that affects users
- consistent analysis must be maintained by examining TLA at same time of year
- whole site is important, not just search
- user privacy
- data classification must be consistent to keep results from skewing
- tools used can be fallible
time is linked with perceptions of value
literature leads you to think that
- there is not a lot of literature
- that children are bold in their search
- teachers tend to get frustrated and bail out more quickly
- both groups excel when tools are designed for them
- don't think the browsing thing is true (browsing is preferred) - someone needs to do a follow-up study on that
Key Ideas
- saving time
- users need a lot of support
What do I want to get out of this data?
- are our guesses on what people are looking for correct
- how are people looking for it
- what are user expectations of what's in the collection
(Michigan Teacher Network's collection development is done by current K-12 teachers and media specialists)
Overall Findings of Study
- 2 words are the mean number of search terms
- perception was that teachers were searching from home, but time of day shows that they are probably doing it at school
Open Questions and Reflections
- most users are looking for curriculum-related materials
- also looking for teacher professional materials
- people are looking for MI city name - are they looking for jobs, schools, tools/resources in the city? what?
- people go directly to the pages that list jobs most frequently, they don't search for it
Top Searches
- "classroom management" is most frequent search
even at different time periods
- pattern emerging, all other searches can be broadly grouped into science
- recurring topics can be seen through the TLA
Two things in mind
- most searching takes place after the school day
- classroom management is a key issue
Science is off the map - top search in curriculum area
- maybe a connection between teachers teaching in science, out of field, and deciding after a terrible day that they never want to have a day like that again
users did not think that they were searching metadata, they thought they were searching resource text
- hidden field was created composed of index of the resource so that users actually were searching over the resource text
some users will type in same term over and over again
users don't always understand that the collection does not contain certain types of information
- users seem to think they're searching the whole internet
Eye Tracking in the Design and Evaluation of Digital Libraries
Bing Pan
many different devices can be used for eye tracking
saccades
scanpaths
Previous Research
- informative areas draw eyes
- different tasks lead to different eye pattern behavior
Study I - Eye Tracking Research on Web Pages
- males fixate more, females saccade more (more scanning behavior)
Study II - Eye Tracking in DL
MetaTest Research
- use and processing of metadata
- image shows that people have reading lulls every 6 or 7 words
- when description and metadata both displayed, people spend time on description
- titles & sources are mostly viewed metadata
Applying Eye Tracking to DL Design and Evaluation
- interface improvement by determining problem areas
Question
Any results by cultural and age differences?
- Study wasn't designed in that way
Question - Manuel
Any difference between those who memorized and those who didn't?
Question - Ann
I expect knowledge differences - understanding what's on the web pages not necessarily
previous familiarity
Question
I expect variability from eye situation - eg glasses
- calibration of each subject's eye occurred, some subjects were not qualified
Discussion