Fennica metadata conversions: statistical monitoring and analysis
1 Preface chapter
This book is a work in progress. More fields will be added to the dataset as they go through data cleaning and harmonization.
Within the FIN-CLARIAH Metadata Harmonization and Analysis work package, we use the Finnish national bibliography (FNB) Fennica to build a harmonized dataset. It serves research and lays the groundwork for further infrastructure work, and it supports the DHL-FI project funded by the Research Council of Finland. Fennica holds metadata for over a million documents, including books, newspapers and maps, with records from 1488 to the present. For more about Fennica, see the National Library of Finland website.
The harmonization runs in two pipelines: the Complete FNB Pipeline and the 1809 to 1917 Period Pipeline. The chapters of this book cover the individual metadata categories.
1.1 The harmonized dataset
The choice of fields to harmonize is guided by the needs of the Digital History for Literature in Finland (2022-2026) project funded by the Research Council of Finland. The project uses digital collections and methods to expand the prevailing understanding of Finnish literary history.
The current release covers the complete digital Fennica, 1,318,062 records. From the MARC 21 source data, 19 harmonized analytical fields were produced. They are distributed over 92 columns, three of which are record identifiers. Most fields keep the source value alongside the harmonized value and, where relevant, add a primary-value column, a multiple-value indicator and derived variables. Publication language, for example, occupies 12 columns: the combined value and its two sources (041,a and 008/35–37) each come with these companion columns.
1.1.1 Fields and columns
| # | Field | Source | Columns | n |
|---|---|---|---|---|
| – | Identifiers | – | id, id2, data_element |
3 |
| 1 | Record type | Leader/06 | record_type |
1 |
| 2 | Bibliographic level | Leader/07 | biblio_level |
1 |
| 3 | Publication status | 008/06 | publication_status |
1 |
| 4 | Genre (008/33) | 008/33 | genre_008 |
1 |
| 5 | Main author | 100 | author_name_100, first_100, last_100 |
3 |
| 6 | Other contributors | 700 | author_orig_700, author_name_700, first_700, last_700 |
4 |
| 7 | Author authority attributes | Kanto (Asteri) | author_birth, author_death, author_age, n_authors, author_keys, author_names_all, author_genders_all, author_births_all, author_professions_all, n_female_authors, n_authors_with_gender, author_key_100, author_name_db_100, author_gender_100, author_birth_100, author_death_100, author_profession_100 |
17 |
| 8 | Title | 245,a–b | title, title_remainder, title2, each with _length and _word |
9 |
| 9 | Publication language | 041,a; 008/35–37 | language_full, language_041, language_008, each with _primary, _multi and lang_*_orig |
12 |
| 10 | Original language | 041,h | language_original, language_original_primary, language_original_multi, lang_original_orig |
4 |
| 11 | Publication time | 008/06–14 | publication_year_from, publication_year_till, publication_year, publication_decade |
4 |
| 12 | Publication place | 260,a; 264,a | publication_place |
1 |
| 13 | Publication country | Geographic linkage | publication_country |
1 |
| 14 | Publisher | 260,b; 264,b | publisher |
1 |
| 15 | Signum | NLF | signum_original, signum_harmonized |
2 |
| 16 | Genre/form | 655,a | genre_655 |
1 |
| 17 | UDC | 080 | udk_orig, udk_harm, udk_aux, udk, udk_primary, udk_multi |
6 |
| 18 | Physical dimensions | 300,c | original, gatherings, width, height, obl, area, gatherings.original, width.original, height.original, obl.original |
10 |
| 19 | Physical extent | 300,a | pagecount, pagecount.multiplier, pagecount.squarebracket, pagecount.plate, pagecount.arabic, pagecount.roman, pagecount.sheet, volnumber, volcount, parts |
10 |
| Total | 92 |
1.1.2 Column descriptions
Identifiers and record-level codes
| Column | Description |
|---|---|
id |
Melinda record identifier |
id2 |
Other system identifier(s) |
data_element |
MARC 008 data element values |
record_type |
Record type (Leader/06), as a descriptive category |
biblio_level |
Bibliographic level (Leader/07) |
publication_status |
Publication status (008/06) |
genre_008 |
Literary form from 008/33, as one of 13 categories |
Authors
| Column | Description |
|---|---|
author_name_100, first_100, last_100 |
Main author (100): full name, first name, last name |
author_orig_700 |
Other contributors as recorded in 700 |
author_name_700, first_700, last_700 |
Other contributors (700): full name, first name, last name |
n_authors |
Number of named contributors (100 and 700 combined) |
author_keys |
Authority keys of all contributors |
author_names_all, author_genders_all, author_births_all, author_professions_all |
Names, genders, birth years and professions of all contributors |
n_female_authors |
Number of contributors recorded as female |
n_authors_with_gender |
Number of contributors with a gender value |
author_key_100, author_name_db_100 |
Authority key and authorized name of the main author |
author_gender_100, author_birth_100, author_death_100, author_profession_100 |
Gender, birth year, death year and profession of the main author |
author_birth, author_death |
Author birth and death year |
author_age |
Calculated lifespan (death year minus birth year); present only when both are known |
Title
| Column | Description |
|---|---|
title |
Title proper (245,a) |
title_remainder |
Remainder of title, subtitle (245,b) |
title2 |
Title and remainder combined |
*_length, *_word |
Number of characters and words in each of the three title columns |
Language
| Column | Description |
|---|---|
language_full |
Publication language(s), combined from 041,a and, where 041 is missing, 008/35–37 |
language_041, language_008 |
Publication language(s) from each source separately |
*_primary |
First (primary) language |
*_multi |
TRUE if the record has more than one language |
lang_*_orig |
Language code(s) as recorded in the source |
language_original (with _primary, _multi, lang_original_orig) |
Original language of a translated work |
Publication
| Column | Description |
|---|---|
publication_year_from, publication_year_till |
First and last year; publication_year_till is filled only for date ranges |
publication_year |
Harmonized publication year |
publication_decade |
Publication decade |
publication_place |
Harmonized place of publication |
publication_country |
Country of publication |
publisher |
Harmonized publisher name |
signum_original, signum_harmonized |
Library call number as supplied by the NLF, and harmonized |
Classification
| Column | Description |
|---|---|
genre_655 |
Genre/form terms from 655,a that match the Finto SLM vocabulary |
udk_orig |
UDC notation as recorded |
udk_harm |
Harmonized UDC notation |
udk_aux |
UDC auxiliary notation |
udk |
UDC classes converted into descriptive categories |
udk_primary |
Primary UDC class |
udk_multi |
TRUE if the record has more than one UDC class |
Physical description (MARC 300)
| Column | Description |
|---|---|
original |
Dimension statement (300,c) as recorded |
gatherings.original, width.original, height.original, obl.original |
Format, width, height and oblong flag as stated in the source |
gatherings |
Harmonized bibliographic format (e.g. 4to, 8vo); 20 categories |
width, height |
Width and height in cm, partly inferred from format |
obl |
Oblong format (1 = yes) |
area |
Page area in cm² |
pagecount |
Estimated total page count |
pagecount.arabic, pagecount.roman |
Pages numbered with arabic and roman numerals |
pagecount.squarebracket |
Unnumbered pages (given in square brackets) |
pagecount.plate, pagecount.sheet |
Plates and sheets |
pagecount.multiplier |
Multiplier applied to the page count |
volnumber, volcount, parts |
Volume number, number of volumes and number of parts (rarely filled) |
1.1.3 Downloads
Download harmonized_fennica.feather.
Harmonized data is also available for a subset of years 1809-1917:
Download harmonized_fennica19.feather.
For more information contact Julia Matveeva (yulmat@utu.fi).
1.2 Metadata availability by field
The share of records with at least one harmonized value varies widely between fields (Figure 1.1, Table 1.2).
| Tier | Field | Source | Records | Share (%) |
|---|---|---|---|---|
| 1 | Record type | Leader/06 | 1,318,062 | 100.0 |
| 1 | Bibliographic level | Leader/07 | 1,318,062 | 100.0 |
| 1 | Title | 245,a-b | 1,318,062 | 100.0 |
| 1 | Publication status | 008/06 | 1,318,010 | 100.0 |
| 1 | Publication language | 041,a; 008/35-37 | 1,317,656 | 100.0 |
| 1 | Genre (008/33) | 008/33 | 1,314,561 | 99.7 |
| 2 | Publication time | 008/07-14 | 1,265,311 | 96.0 |
| 2 | Publication place | 260,a; 264,a | 1,224,289 | 92.9 |
| 2 | Physical extent | 300,a (parsed) | 1,140,335 | 86.5 |
| 2 | Publication country | derived from place | 1,139,850 | 86.5 |
| 3 | Call number | NLF signum | 981,713 | 74.5 |
| 3 | Publisher | 260,b; 264,b | 949,057 | 72.0 |
| 3 | Physical dimensions | 300,c | 868,365 | 65.9 |
| 3 | UDC | 080 | 859,589 | 65.2 |
| 3 | Author names | 100; 700 (authors) | 854,774 | 64.9 |
| 4 | Asteri identifier | Kanto | 854,774 | 64.9 |
| 4 | Author gender | inferred from name | 792,495 | 60.1 |
| 4 | Birth year | Kanto | 465,326 | 35.3 |
| 4 | Profession | Kanto | 353,324 | 26.8 |
| 4 | Genre/form (655) | 655,a (SLM) | 248,020 | 18.8 |
| 4 | Original language | 041,h | 196,600 | 14.9 |
| 4 | Death year | Kanto | 192,554 | 14.6 |
| 4 | Calculated lifespan | derived | 190,555 | 14.5 |
1.3 Metadata availability over time
Availability of authority-dependent and classification fields changes strongly over time (Figure 1.2). Decades with fewer than 100 records are omitted; the shaded band marks the Grand Duchy period (1809–1917).
| Period | Records | Author names | Asteri identifier | UDC | Genre/form (655) | Physical dimensions | Original language |
|---|---|---|---|---|---|---|---|
| before 1809 | 43,533 | 55.1 | 55.1 | 46.1 | 34.3 | 50.7 | 3.2 |
| 1809–1917 | 80,636 | 61.9 | 61.9 | 41.9 | 24.0 | 41.1 | 16.7 |
| 1918–1944 | 114,407 | 59.7 | 59.7 | 30.2 | 12.3 | 38.0 | 12.6 |
| 1945–1999 | 521,977 | 54.4 | 54.4 | 75.5 | 9.4 | 78.4 | 15.1 |
| 2000– | 504,758 | 76.6 | 76.6 | 74.2 | 28.6 | 68.6 | 16.5 |