Digitalisation has fundamentally changed the public sphere since around the mid-1990s. Large parts of publishing and public communication have shifted to the Internet. Media use – such as the reception of news (Behre et al., 2024; Newman et al., 2023) – is also increasingly taking place online. In contrast, older media, especially printed media, are becoming less relevant. The digital transformation not only affects traditional mass media but goes far beyond this: the whole society is mirrored in the digital public sphere including cultural, political and commercial activities. As a result of digitalisation, the public sphere is also expanding into new areas, such as everyday communication, which has become particularly visible in social media. There are several characteristics which clearly distinguish the digital public sphere from the analogue public sphere and which must be considered when it comes to preserving the digital cultural heritage.

As a consequence of the digitalisation of the publication market and the communication in cultural, political and commercial activities, the collection mandates of national libraries were extended in many countries to comprise digital media to preserve the cultural heritage in the digital age (Altenhöner & Schrimpf, 2014; Schmitt, 2011; Vlassenroot et al., 2019, 2021). This also applies to the German National Library and the law it is based on. According to the German National Library Act (Gesetz über die Deutsche Nationalbibliothek; DNBG), it is the task of the DNB to collect all media works („Medienwerke“). Media works are defined in § 3 (1) as follows: “Media works are all representations in writing, image and sound that are distributed in physical form or made accessible to the public in non-physical form” (translated by the authors). The task is not limited to distinct media. What is required is the publication, i.e. public accessibility of media works, and this includes websites and social media.

The collection mandate (DNB, 2023) applies without restriction: “Our collection mandate aims for completeness, regardless of the form of the media works or the form of publication” (translated by the authors). Completeness might be aimed at regarding distinct media like e-books or e-journals, but it cannot be achieved in the case of web archiving for several reasons. First, the totality of the “German web” is unclear as it comprises much more than the .de domain: all websites with relation to Germany, German language, culture or history as well as websites whose owner’s place of business is in Germany. Besides the DENIC list,¹ it should be hard to get a complete overview of websites fulfilling these criteria, let alone to keep up with the constant change in new websites arising and websites which are taken down. Second, even after identifying a set of relevant websites, to crawl and archive all their versions over time is an impossible task since websites are constantly updated, content is added or deleted without notice and it is impossible to archive all versions (Brügger, 2018).

As completeness of the collection is impossible to achieve in archiving the German web, the DNB is confronted for the first time in its history with the question of how to define selection criteria (DNB, 2016) and to justify them (Altenhöner & Schrimpf, 2014). A key aspect of the collection mandate of the DNB is to collect “without judging content” (Deutsche Nationalbibliothek [DNB], 2017, p. 11) which does not conform with the idea of a content-related selection process. However, it can be interpreted that the selection criteria should be as neutral and consensual as possible and well justified. Guiding principles are the terms “exemplary diversity” and “cultural heritage.” How these principles are to be interpreted must be reviewed permanently, also with the participation of different social groups. The meaning of these principles cannot be definitively fixed. In view of transnationally oriented media such as the web, the limitation of the collection mandate to an individual nation also requires a new, differentiated interpretation.

The motivation for this article is the DNB’s need to massively increase its web archiving activities. Before proposing our guiding principles, we provide a closer reflection on the challenges of archiving the web regarding content selection and technology in section two. Section three then addresses social media as a special case of web archiving. In section four, we introduce our approach for operationalising the guiding principles of relevance and diversity in four dimensions. The section is completed by the technological requirements to address the challenges of the advanced web technology as well as to support a (semi-)automated selection process, complemented by aspects of transparency to be considered in the selection process. Finally, in section five, recommendations for further steps to operationalise the concept of relevance and diversity and how to cope with the abundance of the German web are given.

When full archiving is not possible, decisions about selection processes should be the starting point for building and expanding a web archive. Therefore, this paper focuses on the discussion about how to define processes for deciding what should be archived. Based on these measures could be identified and applied to enable the selection process. It is obvious that a broad expansion of a web archive bears technological challenges for the whole life cycle of web archiving as well as organisational and financial issues, but these must be addressed in a later step after first having defined content related selection criteria. The technological, organisational and financial limitations that will occur when implementing the collection concept will have an impact on the scale, but not on the basic content selection decisions. Hence, in this paper, technological issues are only discussed in the context of enabling the suggested selection criteria while the full operationalisation to address these issues must be discussed subsequently in a different context. The problems and developments regarding crawling, indexing, searching and use of web archives as well as long term preservation and legal issues (e.g., data privacy and copyright) are out of scope of this paper.

The DNB’s mission is to archive everything that was published in Germany, in German language and about Germany. With traditional publications these could be matched to selection criteria that would work in a setting of formal publishing cultures. However, on the web, publishing does not follow formal and standardised procedures. Therefore, a new selection strategy needs to be defined that takes into account content-related criteria. Regarding print media the DNB applies formal criteria like a minimum circulation figure or a minimum volume to decide whether a publication is ingested in the collection. In the case of web archiving the selection criteria cannot be determined purely formally as the characteristics of the web as media differs significantly from print media as is shown in the list below, hence criteria must be defined content-related. The selection must be based not only on the mission of archive libraries to preserve cultural heritage for the long term, but also on the current user demand. The question of expectations of future generations can only be answered hypothetically. These selection criteria must be justified, made transparent and be revisited regularly.

The digital public sphere has characteristics that make the systematic description and selection of web offers for archiving difficult. This results in a number of challenges for the DNB:

Overall, the digital public sphere is a comprehensive, less structured representation of society as a whole, containing various modes of access to the world (knowledge, entertainment, culture, advertising and other forms of persuasive communication, everyday communication).

A further basic part of the definition of selection criteria is the decision whether basically everything on the (national) web may be collected or if certain categories of content must be excluded. This question is mainly relevant for dealing with potentially illegal content. This could be, for example, violence, hate speech and online harassment, or sexual content – depending on respective legal frameworks that are in place. It can be argued that everything on the web can be regarded as published, hence is part of the collection mandate. As legality of the content cannot be examined in detail before archiving, the DNB applies the common rule that it takes down illegal content and excludes it from being indexed, if appropriate, as soon as the DNB is notified of illegal content in its web archive. Generally, everything on the German web is seen as part of the historical tradition and worth to be archived.

Defining selection criteria for a national web archive is first of all content-related, but technological aspects are closely interlinked to content-related decisions, e.g. the appraisal of the relevance of a website or how to deal with personalisation and the dynamics of websites or the use of focused crawlers.

The web has seen a dramatic alteration since the first initiatives for web archiving, specifically the foundation of the Internet Archive in 1996. Consequently, Laursen and Møldrup-Dalum recognised in 2017 that “the archive separates itself increasingly from the live web the archive tries to preserve”. Essential reasons for the divergence between the live web and the archived web are, on the one hand, the development of the web structure and the role of links as well as the continuous dynamisation and personalisation of the web. Both characteristics immediately affect the performances and modes of operation of classic web crawlers, whose selection strategies are only to a limited extent applicable to today’s web.

Hyperlinks were the central element of the Hypertext Markup Language (HTML) and interlinked all available information in the World Wide Web (Berners-Lee, 1999). Web crawlers typically follow this paradigm and traverse hyperlinks to find potentially relevant websites. However, the role of hyperlinks has seen several developments (Helmond, 2019). Initially only being utilised as navigational elements, their distribution was soon used as an indicator of a website’s importance with the PageRank algorithm developed in 1996 as a famous example. Since then, new types of hyperlinks, such as deep links and app links (Helmond, 2019), dynamic links and asynchronous triggered links, have been established. Social media have introduced platform-specific link characteristics like the usage of URL shorteners as well as the focus on intra-website links (e.g., retweets) have led to further fragmentation of the web (Helmond, 2019; Lee et al., 2009). Consequently, the paradigm that each URL uniquely identifies a page on the web, initially only queried via the GET method,² has been softened up.

On top of the link structure, new web technologies have made web archiving more complex. The advent of CSS, Javascript and client-side executed scripts (Ajax) had already been identified as a major challenge to web archiving in 2012 which called for new web page processing in indexing methodologies (Gyllstrom et al., 2012) and JavaScript had been responsible for 33.2% more missing resources in 2012 than in 2005 (Brunelle et al., 2016). Nowadays, websites are typically built in a modular way, consisting of several files and relying on complex server-side processes and user interaction which require server-side preservation strategies for preservation of their contents (Bočytė & de Vos, 2018).

Web crawlers have only partially been adapted to these changes of the web structure. Focused crawling techniques aim to identify web pages related to a topic of interest, but they require the identification of topic-relevant web pages, which has become increasingly inefficient and noisy as the complexity of the web has grown (Aggarwal, 2019). While Google’s web crawler has allowed the use of POST requests since 2021,³ there remains an open question of which requests are suitable for web archiving. Also, the web archive should reflect the structure of the web since users want to conduct analyses of the granularity of single websites and the structure of the web in general (Holzmann & Nejdl, 2021). Hence, new web crawlers need to drift away from the original concepts of the web and instead find new solutions that enable them to extract and store as complete information as possible from the contents of web pages.

Due to the personalisation of the web, two users can see different versions of the same web page at the same time.⁴ Kelly et al. (2013) have identified different cases potentially leading to such a situation, including different website layouts (e.g., mobile vs desktop mode) and local website content based on the user’s IP address or GPS position (e.g., local news and weather forecasts). Exceptional cases of personalisation are advertisements and social media: recommendations for posts and products or displayed trends can be tailored to user groups, and different user groups might be shown different content via microtargeting for political influence (Ribeiro et al., 2019).

To deal with personalised content, web crawlers can either attempt to disable any personalised information, e.g., by declaring themselves as bots in the request header, or by taking the role of different personae to retrieve different views from the same website. In the former case, the challenge lies in the definition of user profiles and the identification of websites that actually offer a relevant diversity in information regarding personalisation. When viewing a snapshot in a web archive, users should get access to metadata specifying which personalisation is behind the respective stored snapshots (Kelly et al., 2013).

Limited resources are one of the main reasons why the web cannot be archived in its entirety. Lee et al. (2009) define three dimensions of resource requirements when creating web archives:

While we do not yet have exact ways to measure the size of the German web, some data may help to illustrate the potential dimensions: Figure 1 is based on the analysis of the content of .de snapshots in the Internet Archive between 1994 and 2013 performed by Holzmann et al. (2016), and highlights the challenges regarding the size of the German web offers with more than 5 billion snapshots until 2013 including 400 million images. Both numbers undoubtedly have increased substantially in recent years.

Relevance is measured in approaches of communication studies via operationalisations of visibility and influence in the public sphere (as a concept see Schatz & Schulz, 1992). Indicators for visibility can be, for example, the usage of offers, number of citations and mentions of actors, and the number of contributions to a topic. Indicators for influence can be agenda-setting effects and opinion power ascribed to (“leading”) media and actors like politicians in high positions. A special approach to define the relevance of topics is the “news value” (Galtung & Ruge, 1965).

Relevance in the public sphere can be supplemented by relevance indicators defined from the perspective of functional subsystems of society (such as politics, economy, science, art, or sports). These subsystems have their own measures of importance. For example, prominence of scientists in public often differs from their reputation in the science system itself. The system-specific relevance becomes obvious through the roles of the actors in the system’s own hierarchy (e.g., the chancellor in the political system) or the results of internal recognition mechanisms (e.g., awards). Following Bourdieu (1985), the definition of relevance at the heteronomous pole of a field is oriented toward outside influences and audience tastes, while at the autonomous pole relevance assessment follows internal standards of quality and self-defined societal goals of professions.

Other forms of social differentiation also have an impact on relevance assessments, e.g., regions, milieus, or classes. Therefore, relevance must always be defined in a distinct perspective. This can lead to a variety of different definitions of relevance. For the purpose at hand, it is important to follow the most consensual and stable relevance assessments possible.

Diversity as a norm means that all interests of population groups and all manifestations of society shall be represented in the public sphere, regardless of the quality, visibility or influence of these interests and manifestations. Diversity can refer to different dimensions of content, including events, topics, opinions, actors, spaces, and formats. The norm of diversity has been intensively discussed and empirically investigated in the context of media and public sphere (e.g., Hendrickx et al., 2022; Joris et al., 2020; Loecherbach et al., 2020; Napoli, 1999) to which the DNB can refer.

The operationalisation of diversity as well as of relevance depends on several normative decisions. In the first step, dimensions of diversity must be defined for both sides: for society’s representation and preferences of audience groups. Then, in the second step, the different manifestations must be found in one dimension, for example, forms of everyday culture in different milieus or different opinions on a given issue. Ultimately, it is a question of capacity, determining how much diversity can be documented and archived, since the manually curated selection of diverse material is complex and time-consuming. Relevance and diversity can be operationalised meticulously and with high complexity which detracts the practical manageability and raises many questions of detail. A compromise between a documented rationale and practicability needs to be found.

Regarding relevance and diversity the following four dimensions need to be considered.

Manually curating selection processes require high personal resources. Therefore, technologies that can support or conduct the selection process will be needed to fully implement the selection based on relevance and diversity.

Where archiving of the full (national) web is not possible and selection criteria need to be applied by an institution, transparency about the selection criteria that have led to the inclusion or exclusion of websites can directly impact the perceived quality of the archive. Selection criteria directly influence the composition of the collection and therefore become an integral part of the archive themselves. Only if selection criteria are documented and possible inappropriate content is defined, users will be able to comprehend why specific contents are included and what might be missing. Transparency is particularly important for sensitive topics, for example, when preparing a topical collection on LGBTQ+ websites: users of the archive could receive information on why certain websites were included. This can help to avoid “blind spots” (Donig et al., 2023) and to reflect on potential biases that may still be inherent to the collection.

Apart from transparency on selection criteria, quality-control should also respect aspects of transparency on technical dimensions. Maemura et al. (2018) define the following three elements of provenance or transparency criteria to document a specific crawl:

Selection criteria can be presented in general for the complete crawl, or individually for each crawled website, e.g., as part of their metadata.

A mix of curational and automatised measures and sources of information should be taken into consideration to cater for a good coverage of relevance and diversity. A selection of such measures is presented in Table 1, also summarising the automatised measures as outlined in chapter 4.3.

Apart from the content-related selection process guided by such measures, alternative representations of web content in web archives need to be discussed to bring the content of web archives closer to the original user experience. Hence, there is a need to question the following three aspects:

As the next steps to establish a semi-automated selection process, we envision the following activities:

Having outlined the necessary next steps towards selection criteria for archiving the German web, we aim to elaborate them further in collaborative projects. As future work, concrete operationalisations should be explored, implemented and evaluated, also considering scalability and resource-efficiency. Balancing out the aim for completeness against the current restrictions and limitations requires solutions that facilitate the necessary selection process in a transparent and understandable way. Our concept for using relevance and diversity as main selection criteria is the proposed way of addressing this tension.