INTRODUCTION

Library networked statistics and performance measures are important indicators of the use, uses, and users of networked services that libraries offer their patrons. Access to data that identify what networked services are being used by whom and when can:

• Enable local library directors to compete for resources with other local organizations by documenting the range, extent, and impact of library-provided networked services.

• Assist libraries make a strong case for national, regional, or local community support for technology and information infrastructure by documenting their Internet-based services and resources.

• Facilitate the transition from traditional library use measures such as circulation, reference transactions, interlibrary loans, etc., to network measures that describe the nature and use of library-based network activities and resources.

• Assist libraries in administrative, management, and planning activities.

• Allow individual libraries, regions, and states to compare effectively themselves to others in terms of Internet development, costs, provision of services, connectivity, and use.

• Provide libraries a means through which to assess, compare, and make decisions regarding collections resources in general and future collections development decisions in particular.

Individually, these factors provide libraries with the ability to incorporate network usage data into key decision making processes and planning activities. Together, through a systematic approach to the collection and reporting of library network statistics on a local, regional, national, and international scale, these factors provide significant data regarding use and usage trends throughout libraries that inform policy makers, researchers, and library professionals as to what types of network activities are occurring in libraries and how those uses can lead to library role changes in the networked environment.

Background

With the assistance of a 1998 National Leadership Grant from the U.S. Institute of Museum and Library Services (IMLS), the author (and others) conducted a study to develop national (United States) public library network statistics and performance measures. Through this study, the author developed, defined, and field-tested a series of public library network statistics and performance measures for ultimate collection at the national level (see Figure 1). Overall, the statistics and measures developed in this study provide the means to track various public library networking capabilities and activities such as the type and level of Internet connection; types, public access workstations; available network-based services, such as databases, online reference, and web-based services (e.g., digital libraries); and staff and user instruction.

Upon completion of the network statistics and performance measure project, the author (and others) received a 2000 National Leadership Grant from IMLS to assess the feasibility of and develop a national model for collecting public library network statistics and performance measures. Partners in the current study include the U.S. National Commission on Libraries and Information Science (NCLIS) and the National Information Standards Organization (NISO). Based on the project findings and model field test (discussed in more depth below), the researchers will recommend to IMLS, the public library community, state library administrators, and others, possible approaches to the collection of public library network statistics on a national scale. These recommendations will be included in the final project report to IMLS in May 2002.

At present, the author (and others) is also involved in the Association of Research Libraries (ARL) E-metrics project. This project, begun in July 2000, seeks to identify, define, and standardize a set of network statistics and performance measures for ARL libraries. The intent is to develop a core set of network statistics and performance measures through which ARL libraries can manage their networked resources and services, plan future network resources and services, and benchmark themselves against other ARL libraries. In ad-dition, the project seeks to provide network statistics and performance measures in an outcomes-based context. The project completed field-testing a set of network statistics and performance measures in June 2001.³

This article focuses on three efforts to develop and standardize library network statistics and performance measures. In particular, the article discusses, compares, and contrasts selected aspects of the International Standards Organization (ISO), U.S. public library network statistics, and Association of Research Library (ARL) efforts. The three approaches attempt to capture, describe, and present library networked activities in similar ways through similar approaches – yet they differ in key areas. It is important to note that there are a number of national and international efforts underway that continue to research the library network statistics and performance measure environment. This article summarizes, but does not review, selected efforts presented in Figure 2.

DEVELOPING LIBRARY NETWORK STATISTICS AND PERFORMANCE MEASURES

There are a number of network statistics and performance measurement development, collection, and reporting issues that require resolution (Bertot, McClure, and Ryan, 2000). Among these issues are the:

• Range of Sources of Network Data within Individual Libraries. No two libraries have the same information technology infrastructure, configuration, or systems implementation. This creates a substantial challenge for the collection of the same data from libraries using similar (but different) technology in various configurations.

• Data Quality. Librarians want accurate, credible trustworthy, valid, and reliable data that describe the use and uses of their networked resources and services. However, there is a false expectation that machine-generated or captured data (e.g., online database sessions, web visits) are exceptionally accurate. This is not the case, as the quality of such data depends on a number of factors. Indeed, Figure 3 demonstrates the difficulty of capturing a simple database session count. All collected and reported data related to libraries and services are best seen as estimates – even those that are currently collected (Library Research Service, 1995).

• Limits to Longitudinal Data. Longitudinal data are useful to track trends within a library and as a check for unusual spikes or bad data. But the rapidly changing nature of information technology will have a substantial impact on the life cycle of the network statistics and performance measures. It is unclear as to just how longitudinal network statistics will be given the need to change what they capture and how as technology changes.

• New Data Collection Techniques. Network measures require researchers and professionals to consider the benefits and/or necessity of using new data collection techniques including traditional quantitative methodologies (surveys, or Likert scale surveys of user satisfaction with network services) in new ways; less familiar qualitative (e.g., focus groups, interviews) methods; adapting traditional methodologies (e.g., pop-up Web-based surveys); and creating new methodologies (e.g., Web-based transaction log analysis) to capture network usage data. In some cases, to promote timely and responsive measures it may make sense to rely on carefully developed samples at the local, state, and national levels rather than 100% population responses. In other cases, sequencing data collection, in which a question is not asked annually but every two or three years may be appropriate to reduce local data collection burden.

• Ability of Local Libraries to Collect Network Measures. In order to attain national network statistics and performance measure data, it is necessary to collect the raw data at the local library outlet level. It remains unclear as to the ability of the library outlets to collect such data.

• Preparation and Training Necessary. Collecting data on network measures will require preparation and library staff training to be successful. There are a number of training topics that need attention including the identification of the range and diversity of technology generating network measures; the notion that at least for the near term estimates, samples, and the lack of long term longitudinal network data may be the norm; and introduction to new data collection techniques and how they may be applied to collecting network measures relevant to local libraries.

• Training in New Data Analysis Techniques Necessary. Training in how to analyze and interpret these new network measures (some more than others) will be necessary at all levels. For example, training librarians to download pre-formatted data into a standard spreadsheet and then do some basic analysis. In addition, few of those interviewed outside of some systems librarians knew how to effectively use network analysis data. In the case where the library has systems staff, network data may only be used for internal technical purposes. But often, these technical experts have not seen the utility of this data for wider administrative purposes such as demonstrating use, showing need, garnering funding.

• Training in the Analysis and Use of the Network Data Reported Necessary. Librarians have spent decades convincing governing boards and administrators that circulation counts, attendance records, reference transactions, etc. that go up annually are a „good thing.” Now that these and other traditional counts are stagnant or declining in many cases, librarians have to re-educate governing boards that web visits, electronic reference questions, full text downloads, and other indicators are as or more important than the traditional measures.

• Partnerships – Maintaining Control, Obtaining Data. Partnerships, both formal and informal, are a way of life for libraries. But they can create problems for the collection of needed network measures when the technology, network, or databases are not owned by the library (i.e., online database vendors, consortia). It is important for libraries to foster reporting agreements with external entities to ensure that libraries receive use data for services to which they subscribe or can access through subscriptions by other entities (e.g., government agency, consortia).

These issues, at a minimum, require attention and an acceptable level of resolution for it to be possible to develop and collect national and international library network statistics and performance measures.

Figure 3: Capturing an Online Database Session.

SELECTIVE COMPARISONS FOR NETWORK STATISTICS AND PERFORMANCE MEASURES

Although there are a number of efforts to develop library network statistics and performance measures (see Figure 2), at their core these efforts seek to understand the use, uses, and impacts of library networked services on the libraries that provide them as well as the communities that they serve. These efforts, at least in part, also attempt to provide libraries with a means through which to benchmark their networked services within and across libraries. In essence, these efforts seek to provide the answer to a series of critical questions:

• How well are we [the library] providing networked services to our users?

• How are the services we provide being used by our service population?

• Which services, and with what frequency, are patrons using?

• What is the „right mix” of traditional and networked services for our library’s service population?

The network statistics efforts described in this article provide a first step towards trying to answer this key questions. They do not, however, provide all the answers and, in many cases, the answers will only come with the adoption of these statistics and measures into general library planning, evaluation, and decision-making efforts.

The ISO, U.S. Public Library, and ARL Efforts

Together, the ISO, U.S. Public Library, and ARL network statistics and performance measure efforts represent key research activities in the development of library network statistics and performance measures. While the U.S. Public Library and ARL efforts are limited to single library types (public library and ARL academic library members, respectively), the ISO effort crosses library types.

A comparative discussion of all aspects of the initiatives is beyond the scope of this article. Thus, this section concentrates on the difference in approaches to measuring similar library networked services and resources that the projects take. As Figure 4 demonstrates, there are subtle differences in approaches to similar network statistics definitions and measurement that make the definitions incompatible. The statistics also demonstrate different emphasis on the types of library services and resources use and uses that libraries desire to capture.

For example:

• Sessions. Though they use different language, the three efforts essentially view a session to an online database as a count of log-ins to the service.

• Documents or records viewed. Here all three efforts differ in approach. To some extent, this reflects the difference in library type – public libraries are not necessarily as interested as academic libraries in the distinction between a citation or abstract view. The U.S. public library and ARL efforts emphasize online database services and do not include OPAC record views as does the ISO approach. Also included in the U.S. public library and ARL approaches are saving, e-mailing, or otherwise user access to the material.

• Downloading transactions. ISO considers this a separate user action from that of a record/document view. It is problematic, however, to capture printing activities as printing is usually a local function that database vendors will not be able to count.

• Connect time. Again, this is an ISO statistic that neither the U.S. public library nor ARL efforts consider.

• Number of queries and Number of full text titles available by subscription. These statistics are considered by the U.S. public library and ARL efforts, but not the ISO effort.

• User location. The U.S. public library context considers two user database access locations – in-library and remote. The ARL effort considers three access points – in-library, outside the library but within the institution (e.g., a campus department), and remote. ISO considers three locations – in-library, outside the library but within the institution (e.g., a campus department), and outside the institution (remote) but within the population served. For their to be a reconciliation in the user access locations, the ISO definition needs to incorporate a fourth category of remote and not in the service population.

A detailed comparison of all proposed network statistics across the three efforts would provide additional differences. The above, however, serve as ex-amples of differing contexts, emphasis, and philosophy.

Reconciling the Differences

Reconciling the different approaches, definitions, and approaches to measuring library network services and resources will require collaboration and cross-fertilization of the efforts. Fortunately, the U.S.-based National Information Standards Organization (NISO) has rejoined the ISO library statistics effort after a near two-year absence. U.S. delegates to the ISO Technical Committee 46 (Information and Documentation), subcommittee 08 (Statistics and Performance Evaluation) include representatives from both the U.S public library and ARL communities.

In addition, the U.S., though NISO, is undertaking a revision to its Z39.7 Library Statistics standard. It is clear that these efforts – the ISO TC46/SC08 and NISO Z39.7 – need to adopt standard definitions, language, and approach where possible. There will be differences in emphasis of service and resource measurement that necessarily reflect differing contexts. This is to be encouraged so as to encourage relevant and meaningful measures to the represented communities. It would be unfortunate and counterproductive, however, to travel separate routes for statistics that are quite similar in intent and definition. In the case of online database vendor statistics, the implications of different standards could be quite problematic as it would require vendors to have multiple use data capturing and reporting mechanisms that may be in-compatible.

NEXT STEPS

There are a series of next steps on both the U.S and international fronts that bear mention here:

• NISO/ISO library statistics. Both NISO and ISO are reviewing their library statistics standards. NISO is only at the beginning of its five-year review process whereas ISO is in the final stages of its revisions. There is continued work in this area, however, and ample opportunity for integration and standardization.

• U.S. public library network statistics activities. The U.S. public library effort is moving towards creating a national network statistics collection and reporting system. Indeed, in October 2001, the project will undertake a field test of a national collection and reporting system in which ten states (Alabama, Colorado, Georgia, Illinois, Kentucky, Louisiana, North Carolina, Ohio, Pennsylvania, Texas, Virginia, Wisconsin, and Wyoming) will participate. The goals of the field test are to

1. Create a fast response approach to the development, collection, analysis, and reporting of network statistics and performance measures;

2. Foster an environment of constant change;

3. Implement a reasonably burden free data collection and reporting process for public libraries, state library agencies, and library consortia;

4. Work with non-library partners to gain access to library network data (e.g., vendors, consortia, state library agencies); and

5. Produce national estimates of public library network service uses and usage.

• ARL network statistics. The ARL research effort continues its work on network statistics and measures. As of June 2001, the study completed its field test of library network statistics and performance measures. The effort resulted in the development of a manual to assist libraries in their collection and reporting of network statistics and performance measures (Shim et al., 2001). The study will move into an outcomes phase that examines network statistics and performance measures in an outcomes context. The goal is to develop an outcomes framework for network statistics and performance measures.

Regardless of how these efforts progress – preferably in tandem – there are a number of issues that require consideration by the library community at large regarding the development, collection, and reporting of network statistics and performance measures. These issues are discussed below.

ISSUES IN THE NETWORK STATISTICS ENVIRONMENT

When considering library network statistics and performance measures, there are a number of considerations for libraries, administrators, policy makers, and researchers:

• Library culture of assessment. In addition to having a systematic approach to network statistics and performance measure activities, libraries will need to adopt an overall culture of assessment. Lakos defines a culture of assessment as (1999, p. 5):

  „The attitudinal or institutional changes that have to occur in order for library staff to be able to work in an environment where decisions are based on facts, research and analysis, and services are planned and delivered in order to maximize positive outcomes and impacts for the library clients”.

  As such, libraries need to focus on a systematic approach to the assessment of library services, resources, and initiatives in order to better understand the impact of those services, resources, and initiatives as well as to undertake changes and/or modifications to best meet the needs of library users. While libraries undertake a number of assessment activities, they are not, in general, part of a systematic evaluation and assessment process that permeates the library. It is in such an assessment framework that network statistics and performance measurement activities need to reside.

• Library data collection, analysis, and presentation management system. U.S. public library and ARL studies demonstrate that libraries overall do not engage in a systematic and focused data collection system. Few, if any, have staff specifically responsible for library-wide data collection, analy-sis, or reporting and presentation efforts. Moreover, library staff may not be trained appropriately in the various methodologies, data analysis techniques, and reporting procedures required to engage in statistical and performance measurement activities. The data collection situation is particularly problematic in the networked environment, as data collection efforts in this area require additional technical and research skills.

• Library staff development and training. Given the culture of assessment issue, combined with a general lack of systematic network statistics and performance measurement activities in participating libraries, it is clear that there is a need for staff development and training in both assessment and network statistic activities. This training should incorporate an overview of the benefits and impacts of evaluation activities; the value of evaluation in decision making and resource allocation processes; network statistics and performance measure definitions, collection activities, methodologies, and reporting systems; and the incorporation of network statistics and measures findings into decision making and resource allocation activities.

• Network planning and evaluation activities as a stand-alone process. While it seems obvious, library network activities reside in a larger organizational context. Despite varying local factors, it is important to consider the planning and evaluation of library networked resources and services as part of larger organizational planning and evaluation activities. For example, the decision to subscribe to various online databases needs to occur in the larger decision regarding library collection development efforts.

• Develop multi-agency reporting systems. It is clear that libraries do not control the use data for all networked services and resources. This is demonstrated by online database vendor statistics. Libraries that receive network services and resources from other entities such as state library agencies or regional consortia, however, also need to work with those administrative entities for usage reports to get a better sense of the overall use of, for example database services, from a library. In such cases, usage reports will go from the vendor directly to the subscribing entity – not necessarily all the participating members. Thus, there is a need to develop a reporting structure that goes beyond the library in such cases. It is also necessary to construct agreements to encourage individual library stats and work with consortia groups to generate meaningful reports for members, etc.

• Investment and/or modifications in infrastructure. Network statistics and performance measures are dependent on the information technology (IT) architecture of a library, consortia, vendor, or other service provider from which the library derives service. If a library finds certain statistics of interest and/or importance, it may require investment in an IT infrastructure that enables the collection of such statistics. Along with that investment may require IT and library staff training in order to understand the configuration as well as the statistics enabled through such a configuration.

• Knowledge of what entity captures what data. Related to the IT architecture issue is understanding what entity (e.g., library, database vendor, or other government agency) is responsible for capturing desired data. Going back to Figure 3, which described a database session, it may not always be the case that the vendor captures unique sessions to the service the way in which the library would like (i.e., with IP address breakdowns). For example, should a database service be accessed through a proxy server or firewall – as in the case of many libraries – the vendor will only capture the number of sessions through that server or firewall with a single IP address. For the library to receive any breakdowns by IP address, it will need to conduct Web log transaction analysis for accesses to the server and/or firewall. In this case, the burden for collecting sessions by such breakdowns as IP address shifts away from the vendor to the library.

• Development of an international reporting system. It is one thing to participate in discussions of network statistics and performance measures standards in terms of definitions and measurement. It is another to engage in international data collection and reporting activities. While there are international reporting systems (e.g., LibEcon), for some library statistics, there is not such a system for the collection and reporting of network statistics. Such a system, however implemented, will need to

1. Adopt standard terms, definitions, and operationalization of network statistics and performance measures (on-going through the ISO effort);

2. Create a fast response approach to the development, collection, analysis, and reporting of network statistics and performance measures that reflects the environment in which network statistics reside;

3. Foster an environment of constant change that enables the creation of new, modification of existing, or retirement of irrelevant network statistics and performance measures;

4. Implement a reasonably burden free data collection and reporting process for libraries, agencies that provide library database services (e.g., state library agencies), and library consortia; and

5. Work with non-library partners to gain access to library network data such as online database use statistics and web site use statistics (for libraries that outsource their web site services).

The extent to which it is possible to create such a system remains unclear at this time. Existing local, regional, and national data collection systems are having difficulty in adopting and reporting proposed network statistics – much less an international system.

These issues point to the need for additional understanding of the network resources and services measurement environment, agreement on the scope and intent of library network statistics and performance measures, and the ex-ploration of the extent to which there is a core set of library network statistics that cut across libraries regardless of library type. The issues raised also indicate a need for libraries to enhance librarian technical, evaluation, planning, and methodological skills to better engage in evaluation activities in general and the networked environment in particular.

ACKNOLEDGEMENT

The author wishes to acknowledge the research efforts of Charles R. McClure, Joe Ryan, Denise Davis, Wonsik „Jeff” Shim, and Bruce Fraser on portions of the various research initiatives discussed in this article.