ࡱ> _a^@ 1jbjb  lhh/+rl<<<<<<<8,@:z(999, <9"99u<<u@uuu9X<<uP66<<<<9uu{V@<<4 c Q" .x{u{uWorkshop on Web Metrics in NSDL August 2-3, 2004 Workshop Summary The NSDL Web Metrics initiative is trying to answer the question of how people are using NDL and how the usage grows and changes over time. NSDL will perform a new Web Metrics pilot, with all pilot sites using the same commercial tool, to determine the validity of the page-tagging method. Background NSDL has been growing over the past 4 years; additional projects are added every year, more resources are added to the main portal, NSDL.org, every month, and 2004 brings the addition of a new type of project, Pathways Projects, which are responsible for targeting specific NSDL audiences by creating a website for those audiences. To achieve its educational goals, NSDL must assess its use and effectiveness. Over the past few years, the field of web analytics has matured to a point where user behavior can be analyzed in an effort to improve a website, the users experiences of that website, and to help a company become more effective. Web metrics can be dissected in many ways to produce a sophisticated level of information. However, analyses must be performed to create useful information and act on that information. These tasks must still be performed by people, and a strategy must be developed to successfully utilize web metrics. Two types of web metrics tools exist, those that use webserver log data (log aggregation) and those that use page-tagging. In February 2002, the NSDL Educational Impact and Evaluation Standing Committee (EIESC) decided to attempt to answer the question: How are people using the Library? A web metrics pilot was undertaken in April, May, and June 2002. Six pilot sites participated by providing a standard set of information from their webserver logs. [See  HYPERLINK "http://eduimpact.comm.nsdl.org/events/?pager=90" http://eduimpact.comm.nsdl.org/events/?pager=90 for more information on the 2002 pilot.] The pilot study proved difficult because it required a lot of time, especially for initial setup, to report the desired web metrics. Additionally, data from the six pilot sites was inconsistent due to differences in reporting standards and tools used. The EIESC determined that the costs of the pilot were greater than the information gained. The Committee decided that web metrics can provide useful data but a different approach was necessary. Since web metrics approaches involve technical implementation, the NSDL Technology Standing Committee (TSC) joined the EIESC to help develop a web metrics strategy for NSDL. [See  HYPERLINK "http://eduimpact.comm.nsdl.org/events/?pager=225" http://eduimpact.comm.nsdl.org/events/?pager=225 for more information on the joint Standing Committee effort.] Detailed Workshop Summary The NSF NSDL Web Metrics Workshop is working to answer the question of how people are using NSDL and how that usage grows and changes over time. The workshop was attended by 23 members of the NSDL community, 2 representatives from NASA, and Paul Strupp, an invited expert on web metrics. Workshop participants initially began to enumerate the desired metrics for NSDL to measure. Such metrics include use of search, clickstreams between and through NSDL sites, and number and frequency of return visitors. However, after initial discussions, participants decided to forego the creation of such a list for discussions of desired effects and requirements for web metrics in NSDL. Participants believe it is important to use web metrics to supplement other NSDL data. Thus, certain NSDL data needs to be documented and maintained. A list of such NSDL data will be compiled at a later date. Examples include the number of items accessible through NSDL.org and NSDL partner sites, the number of search engines that have indexed the site, and the movement of content via OAI within NSDL and harvested from NSDL partners. This NSDL data will aid in the analyses necessary to draw meaningful conclusions from web metrics. In addition to the availability of NSDL data, workshop participants highlighted the importance of maintaining a knowledge bank about NSDL users to aid in analyses and interpretations of web metrics. This list will also be compiled later, but examples include use cases, user profiles, models of successful and failed visits, and site classifications (the relative importance of each page). Three high-level NSDL goals were addressed at the workshop in terms of using web metrics to analyze the achievement of the goals. Many potential metrics, issues, solutions, and analytical options were proposed through the discussions of achieving these goals. For example, number of downloads, Bobby-compliance, number of unique users, and location of users. Goal 1 - High quality learning resources are accessible (part 1 of the sentence completed by Goal 2) Goal 2 - accessible to a large spectrum of the US population (part 2 of the sentence started by Goal 1) Goal 3 - There is value added for users and projects to participate in NSDL. Business Requirements for NSDL Web Metrics A list of business requirements for NSDL web metrics tools was defined. This list is not exhaustive but should include all of the requirements that would be considered of high importance. total cost of ownership analysis (build vs. buy) data structure that allows for information beyond pages (clicks, db transactions) ease of use maintainability by a central source low barrier to entry easy for projects to implement ability to analyze search logs by search terms track cross-site traffic track user demographics anonymously privacy maintaining respect for NSDL policies infrastructure independent (browser, server) maintain security of data scalable licensing model (3rd party option) must accommodate evolving NSDL structure and funding mechanism NSDL owning data access to un-aggregated data (non-proprietary format) interactive and API each project needs to see own data & global view customizable reports with customizable access merging other external data (yahoo site statistics) must not interfere with user experience (e.g. site performance, 508-compliance) automated summarization capabilities technology not preclude capturing data from any significant user group establish maximal guidelines for support of this effort by CI and projects and build into RFP extensibility extend data and interfaces transparency of method (confidence in data) develop strategy for using metrics & use the strategy to assess proposed solutions A participant breakout group focused on business requirements relating to search. The requirements are stated in terms of NSDL goals in analyzing search and web metrics to discover user actions and goals. Issues that arose include the need to outline what should be tracked at both the individual project and NSDL-wide level to help guide projects with search analyses and goals. Two breakout groups created a general prioritization of the business requirements. The total cost of ownership requirement is first on both lists. However, further prioritization varied greatly between the two groups. A final breakout group addressed business requirements in terms of participating in a pilot study. Other important issues for a successful pilot study were also outlined along with a plan of action. The 2004 Web Metrics Pilot will be a pilot of the page-tagging method with requires each page to include one line of javascript code so that a users visit to the page can be noted. The pilot should include Core Integration and mature projects along with one or two small sites. The pilot should last for 3-6 months after the time required for deployment. A phased approach should be taken so that simple measures are studied first while adding more interesting questions later. Pilot sites (and any new site implementing a web metrics solution) require an identification of the actual tasks that need to completed to participate. Pilot sites also require a low barrier to implementation. Desired outcomes of the pilot are to show cross-site traffic, comparative assessment of web server logs and page-tagging data to determine confidence in the page-tagging method, an assessment of the transparency of the underlying system, an assessment of the privacy provided to users, filtering local (developer) and robot traffic, and determining the degree of ownership of the collected data for Core Integration and involved projects. Important Business Requirements for the Implementation of a Web Metrics Pilot low barrier to entry easy for projects to implement track cross-site traffic ease of use NSDL owning data transparency of method (confidence in data) each project needs to see own data & global view must not interfere with user experience (e.g. site performance, 508-compliance) track user demographics anonymously privacy infrastructure independent (browser, server) maintain security of data Action Items for NSDL Web Metrics Coming out of the workshop, participants hope for the following actions to be achieved quickly. Develop criteria for pilot site selection. (participants in next web metrics pilot) (participants will receive adequate support for participation) Pilot activities are structured around the 3 described goals. Pilot sites will work with EIESC & TSC to flesh out metrics. Form a small taskforce to work with pilot sites in both the creation/deployment & reporting. (The current web metrics site becomes the site of the taskforce.) The implementation of web metrics within NSDL should meet the outlined business requirements. Finish annotated bibliography, and tell people about it. Prepare a report of workshop for d-Lib & CI. The CI liaison, Casey Jones, will serve as coordinator for the pilot and taskforce. Web metrics data is just one part of understanding usage of the web sites and developing meaningful interpretations and actions. Supporting data, as well as staff to support web metrics, both in analysis and implementation, are also necessary for successful web analytics results. In addition, many projects may need convincing to participate. This will be especially true if people outside their projects have access to their web metrics data. Outside access will be necessary on at least a small scale for successful analyses and interpretations. In the future, project participation in NSDL Web Metrics may require additional data or assistance from NSDL projects. Such requirements cannot be determined at this time. Page-tagging tools are priced based on page views for a site, so the more page views, the more it costs to use a page-tagging service provider. Recommendations to NSF Build into program solicitation that participating in NSDL-wide web metrics is a requirement. Develop a method for encouraging participation in NSDL web metrics for non-NSDL-funded projects. Support for a central analyst within CI is necessary to implement web metrics solutions beyond the pilot. See  HYPERLINK "http://webmetrics.comm.nsdl.org/" http://webmetrics.comm.nsdl.org/ for additional information.  2004 NSDL Web Metrics Workshop Preliminary Report  PAGE 5 /  NUMPAGES 5  013Dhs + , - \ ] ^ 89:jkl yd&>'(((P),z//0000011/101޷ެә޷~ jUmHjnU jU5 B*ph$j5B*CJOJQJUphB*CJOJQJph0JCJOJQJ$jB*CJOJQJUphB*CJOJQJphjB*CJOJQJUph 5OJQJOJQJ CJOJQJ1 123Dfghs  lm  tu >^`> ! 123Dfghs  ,..y/z///P0000/1Q11111 "Bxy -c,FO?m & F ! >^`> >^`>]ef !!d$e$7%8%&&>'t'''''(W((( & F ! & F((((O)P))!*^**[+++,,,..y/z///P0000  & F ! & F !  & F !0/1Q1111111$a$ !  !0111O1o11111111111111111ؽ CJOJQJ0JCJOJQJmH0JCJOJQJj0JCJOJQJU656CJOJQJ5CJOJQJ CJOJQJ / =!"#$% ,, ` g(HH(dh com.apple.print.PageFormat.FormattingPrinter com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.FormattingPrinter com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:58Z com.apple.print.ticket.stateFlag 0 com.apple.print.PageFormat.PMHorizontalRes com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMHorizontalRes 300 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:46Z com.apple.print.ticket.stateFlag 0 com.apple.print.PageFormat.PMOrientation com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMOrientation 1 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:58Z com.apple.print.ticket.stateFlag 0 com.apple.print.PageFormat.PMScaling com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMScaling 1 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:58Z com.apple.print.ticket.stateFlag 0 com.apple.print.PageFormat.PMVerticalRes com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMVerticalRes 300 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:46Z com.apple.print.ticket.stateFlag 0 com.apple.print.PageFormat.PMVerticalScaling com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMVerticalScaling 1 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:58Z com.apple.print.ticket.stateFlag 0 com.apple.print.subTicket.paper_info_ticket com.apple.print.PageFormat.PMAdjustedPageRect com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMAdjustedPageRect 0.0 0.0 3058.3333333333335 2400 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:58Z com.apple.print.ticket.stateFlag 0 com.apple.print.PageFormat.PMAdjustedPaperRect com.apple.print.ticket.creator com.apple.printingmanager com.apple.print.ticket.itemArray com.apple.print.PageFormat.PMAdjustedPaperRect -75 -75 3225.0000000000005 2475 com.apple.print.ticket.client com.apple.printingmanager com.apple.print.ticket.modDate 2004-08-16T16:33:58Z com.apple.print.ticket.stateFlag 0 com.apple.print.PaperInfo.PMPaperName com.apple.print.ticket.creator com.apple.print.pm.PostScript com.apple.print.ticket.itemArray com.apple.print.PaperInfo.PMPaperName na-letter com.apple.print.ticket.client com.apple.print.pm.PostScript com.apple.print.ticket.modDate 2003-07-01T17:49:36Z com.apple.print.ticket.stateFlag 1 com.apple.print.PaperInfo.PMUnadjustedPageRect com.apple.print.ticket.creator com.apple.print.pm.PostScript com.apple.print.ticket.itemArray com.apple.print.PaperInfo.PMUnadjustedPageRect 0.0 0.0 734 576 com.apple.print.ticket.client com.apple.print.pm.PostScript com.apple.print.ticket.modDate 2003-07-01T17:49:36Z com.apple.print.ticket.stateFlag 1 com.apple.print.PaperInfo.PMUnadjustedPaperRect com.apple.print.ticket.creator com.apple.print.pm.PostScript com.apple.print.ticket.itemArray com.apple.print.PaperInfo.PMUnadjustedPaperRect -18 -18 774 594 com.apple.print.ticket.client com.apple.print.pm.PostScript com.apple.print.ticket.modDate 2003-07-01T17:49:36Z com.apple.print.ticket.stateFlag 1 com.apple.print.PaperInfo.ppd.PMPaperName com.apple.print.ticket.creator com.apple.print.pm.PostScript com.apple.print.ticket.itemArray com.apple.print.PaperInfo.ppd.PMPaperName US Letter com.apple.print.ticket.client com.apple.print.pm.PostScript com.apple.print.ticket.modDate 2003-07-01T17:49:36Z com.apple.print.ticket.stateFlag 1 com.apple.print.ticket.APIVersion 00.20 com.apple.print.ticket.privateLock com.apple.print.ticket.type com.apple.print.PaperInfoTicket com.apple.print.ticket.APIVersion 00.20 com.apple.print.ticket.privateLock com.apple.print.ticket.type com.apple.print.PageFormatTicket nCa?3B'/#ӞPNG  IHDR6XPLTEܼeqvҡخQ`2ߔʺve7dPtS䲶SgҢدХ:xLJ:SgxӇؕܢ7tRNSYbKGDH cmPPJCmp0712HsIDAThC}S0w"z{PZW$Mtvnv刵>}&\WYyi9Xc q׵ֱJMJotDgP$TU(gZfR˯.4:?C+>hǼ޿c܊yL{9ؒyP :Ĝ@yju1TGsLwYRyERN5,ĕ3s9~VYL,4sżk^ٹbqqI;2[h6fy d6]C2qfmkheé-/qG BeԙgZcuȧ(œz[de7]ӹd5JhNfyoֵ?7t=hhNfyCD1c wxYʺ`o1D(dD됇@<̯ahqRʍrU> Gz[&Qt0G/!0?i< ouO4Sfb;&&޸: e(Cwt6#Izκ%d Ɍ@Kro&fd!3B4~2::/t$o0*ڂb TxdTEF˺S$u 5oP$wcnE%[ivoyuQMfU1/o쮖лcli]+r8EhH2¢ԼI2cwx3RMj8yW\1[ )-0쿷" wy5ν!zofOƠQ2IENDB`5DyK 0http://eduimpact.comm.nsdl.org/events/?pager=90yK `http://eduimpact.comm.nsdl.org/events/?pager=909DyK 1http://eduimpact.comm.nsdl.org/events/?pager=225yK bhttp://eduimpact.comm.nsdl.org/events/?pager=225DyK !http://webmetrics.comm.nsdl.org/yK Bhttp://webmetrics.comm.nsdl.org/ i4@4NormalCJOJPJQJmH 0@0 Heading 1$@&5<@< Heading 2$@&5CJOJQJ<A@<Default Paragraph Font,@,Header  !, @,Footer  !&)@& Page Number(U@!( Hyperlink>*B*DC@2DBody Text Indent*^*`8V@A8FollowedHyperlink>*B* + F "F!z z z z zc c7(+/gTTqqqt011"(01 !1,\ 9 j **++XXXT[]alnt!l,b$a?3B'/#ӞKf@(    c jA.``nsdlOnWhite\\GR`TGRR \"`t[@*.Xa f r /++BHy~ -0cj ,4FNOX?Kmt ]f>!A!t!y!!!!!!!" "W"\"""""|####/++++:::::::::::::::::::::::::::::::::::::: Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v2.doc Casey JonesLMacintosh HD:Users:caseyj:Documents:Microsoft User Data:Word Work File A_214 Casey JonesLMacintosh HD:Users:caseyj:Documents:Microsoft User Data:Word Work File A_214 Casey JonesMMacintosh HD:Users:caseyj:Documents:Microsoft User Data:Word Work File A_3862 Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v2.doc Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v3.doc Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v4.doc Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v4.doc Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v4.doc Casey Jones_Macintosh HD:Users:caseyj:Documents:_2004:2004_EIESC_webmetrics:WebMetricsWorkshopReport_v4.doc VK^Z+\D2h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo(h ^`OJQJo(h ^`OJQJo(oh pp^p`OJQJo(h @ @ ^@ `OJQJo(h ^`OJQJo(oh ^`OJQJo(h ^`OJQJo(h ^`OJQJo(oh PP^P`OJQJo( ^`OJQJo( ^`OJQJo(o pp^p`OJQJo( @ @ ^@ `OJQJo( ^`OJQJo(o ^`OJQJo( ^`OJQJo( ^`OJQJo(o PP^P`OJQJo(+\ VKZ                           @**4|* d+ @GTimes New Roman5Symbol3 Arial7 VerdanaC#Lucida Grande3Times? Courier New;Wingdings qhˆFˆF҃&b +!>02 @NSF NSDL Web Metrics Workshop Casey Jones Casey Jones Oh+'0  8 D P \hpx'NSF NSDL Web Metrics WorkshopSF  Casey JonesaseNormalo Casey Jones5seMicrosoft Word 10.1@V@tփ@@Fb ՜.+,D՜.+,P  hp  ' NSDL/UCARt+ r NSF NSDL Web Metrics Workshop Title< 8@ _PID_HLINKS'AX!http://webmetrics.comm.nsdl.org/M1http://eduimpact.comm.nsdl.org/events/?pager=225s0http://eduimpact.comm.nsdl.org/events/?pager=90m nsdlOnWhite  !"#$%&'()*+,-./012345689:;<=>@ABCDEFGHIJKLMOPQRSTUWXYZ[\]`Root Entry FKljbData 71Table?{WordDocument lSummaryInformation(NDocumentSummaryInformation8VCompObjX FMicrosoft Word DocumentNB6WWord.Document.8