MARC Rules based mapping#
Introduction#
The mapping rules#
FOLIO stores its MARC-to-FOLIO mapping rules as a JSON object that you can download and edit via:
/mapping-rules/marc-{bib,holdings}
These endpoints supports GET and PUT requests.
You can find documentation for the mapping options in the mod-source-record-manager GitHub repository.
Bib Transformation#
As of version 1.9.0 of folio_migration_tools, we rely on a hybrid approach to migrating MARC-based bibliographic data into FOLIO that puts a lot of the heavy lifting of mapping from MARC-to-Instance on FOLIO’s data import system. Historically, we have maintained our own implementation of FOLIO’s MARC-mapping system in Python as part of these tools, but this functionality has never had full feature parity with FOLIO’s rules processing system. Development in this area will be low-priority moving forward.
By default this tool will perform a basic mapping, including Title, Primary Contributor, and some format information to create a set of instance records with source=FOLIO and a file of MARC21 records that can be loaded via Data Import using the default single-record import import profile (or a copy), to convert them to source=MARC and perform a full MARC-to-Instance map. This process can be automated via a BatchPoster record type (coming soon) or using folio_data_import’s folio-marc-import script.
Configuring Your MARC Bib Transformer Task#
The first step in preparing to transform MARC Bib records for FOLIO is to create a task configuration:
{
"name": "bibs",
"migrationTaskType": "BibsTransformer",
"ilsFlavor": "tag001",
"hridHandling": "preserve001",
"updateHridSettings": false,
"files": [
{
"file_name": "bibs_suppressed.mrc",
"discoverySuppressed": true
},
{
"file_name": "bibs.mrc",
"discoverySuppressed": false
}
]
}
Excluding some MARC records from the output file for Data Import#
You may wish to exclude MARC records for a subset of your migrated bibs from the Data Import file produced by BibsTransformer (eg. lendable equipment records, or other materials that do not require MARC source records). To accomplish this, you can set data_import_marc: false in the file configuration for the specific MARC file containing those records. For example:
{
"name": "bibs",
"migrationTaskType": "BibsTransformer",
"ilsFlavor": "tag001",
"hridHandling": "preserve001",
"updateHridSettings": false,
"files": [
{
"file_name": "bibs_suppressed.mrc",
"discoverySuppressed": true
},
{
"file_name": "bibs.mrc",
"discoverySuppressed": false
},
{
"file_name": "equipment.mrc",
"discoverySuppressed": false,
"data_import_marc": false
}
]
}
MARC records from equipment.mrc will not be saved to the folio_marc_instances_bibs.mrc file produced by this task, but source=FOLIO instances will be.
Posting BibTransformer MARC records#
While folio_migration_tools’s BatchPoster task does not currently support posting the MARC21 records generated by BibsTransformer, the file can be loaded manually, via the Data Import app in FOLIO, or by using the folio-marc-import script available as part of the folio_data_import module:
Example:
folio-marc-import --gateway_url "https://folio-snapshot-okapi.dev.folio.org" --tenant_id diku --username diku_admin --password admin --marc_file_path "iterations/test/results/folio_marc_instances_bibs.mrc" --split-files --split-size 10000 --batch_size 100
MFHD Transformation#
MFHD records are a great way to export records from your legacy system to migrate to FOLIO. However, FOLIO’s support for storing MFHD records in SRS is limited. If you are using these tools to migrate MFHD-based holdings records, we recommend creating source=FOLIO holdings records and not loading the corresponding MARC data to FOLIO
Configure Your MFHD Transformer Task#
The first step in preparing to transform MARC Bib records for FOLIO is to create a task configuration:
{
"name": "transform_mfhd",
"migrationTaskType": "HoldingsMarcTransformer",
"legacyIdMarcPath": "001",
"locationMapFileName": "location_map.tsv",
"defaultCallNumberTypeName": "Library of Congress classification",
"fallbackHoldingsTypeId": "82747568-cdf3-4980-bba2-e5b38950f65b",
"holdingsTypeUuidForBoundwiths": "1b6c62cf-034c-4972-ac80-fa595a9bfbde",
"hridHandling": "default",
"createSourceRecords": false,
"boundwithRelationshipFilePath": "bib_mfhd.tsv",
"supplementalMfhdMappingRulesFile": "custom-marc-holdings.json",
"files": [
{
"file_name": "mfhd_suppressed.mrc",
"discoverySuppressed": true
},
{
"file_name": "mfhd.mrc",
"discoverySuppressed": false
}
],
"updateHridSettings": false
},
For more information on the task configuration options for MFHD transformation, see HoldingsMarcTransformer
Boundwith Mapping for MFHD Holdings (“Voyager-style” boundwiths)#
The MFHD transformer supports migrating boundwith relationships — where multiple bibliographic records share a single physical holdings record — via the "boundwithRelationshipFilePath" configuration key. The boundwith relationship file is a TSV placed in source_data/holdings/ with columns MFHD_ID and BIB_ID mapping each holdings record to one or more bib records:
MFHD_ID BIB_ID
12345 100001
12345 100002
12346 100003
During transformation, the tool creates a copy of the holdings record for each additional bib (instance), setting the holdingsTypeId to the value of "holdingsTypeUuidForBoundwiths" and generating deterministic UUIDs for the copies. The resulting relationship map (boundwith_relationships_map.json) is written to the results folder and consumed by the ItemsTransformer to create boundwithPart records linking items to their boundwith holdings.
Configuration example#
{
"name": "transform_mfhd",
"migrationTaskType": "HoldingsMarcTransformer",
"legacyIdMarcPath": "001",
"locationMapFileName": "location_map.tsv",
"defaultCallNumberTypeName": "Library of Congress classification",
"fallbackHoldingsTypeId": "82747568-cdf3-4980-bba2-e5b38950f65b",
"holdingsTypeUuidForBoundwiths": "1b6c62cf-034c-4972-ac80-fa595a9bfbde",
"boundwithRelationshipFilePath": "bib_mfhd.tsv",
"hridHandling": "default",
"createSourceRecords": false,
"files": [
{"file_name": "mfhd.mrc"}
]
}
Supplemental MFHD Mapping Rules#
As of v1.9.0, folio_migration_tools supports the use of a supplemental MFHD mapping rules configuration, in lieu of either updating the system rules or creating a full, customized copy of the rules (specified by mfhdMappingFileName in the task configuration). Instead, you can create a JSON file with an object containing only the rules you want to add (or modify). The tools will perform a python dict.update() of the full rules file using the file specified in "supplementalMfhdMappingRulesFile". For example, a supplemental mapping to map public and private holdings notes from 852$x and 852$z (using rules for a 952$x and 952$z, see Limitation and Other Considerations below):
{
"952": [
{
"entity": [
{
"rules": [
{
"conditions": [
{
"type": "set_holdings_note_type_id",
"parameter": {
"name": "Note"
}
}
]
}
],
"target": "notes.holdingsNoteTypeId",
"subfield": [
"x"
],
"description": "Holdings note type id",
"applyRulesOnConcatenatedData": true
},
{
"rules": [
{
"conditions": [
{
"type": "trim"
}
]
}
],
"target": "notes.note",
"subfield": [
"x"
],
"description": "Holdings private note",
"subFieldDelimiter": [
{
"value": " ; ",
"subfields": [
"x"
]
}
],
"applyRulesOnConcatenatedData": true
},
{
"rules": [
{
"value": "true",
"conditions": []
}
],
"target": "notes.staffOnly",
"subfield": [
"x"
],
"description": "If true, determines that the note should not be visible for others than staff",
"applyRulesOnConcatenatedData": true
}
]
},
{
"entity": [
{
"rules": [
{
"conditions": [
{
"type": "set_holdings_note_type_id",
"parameter": {
"name": "Note"
}
}
]
}
],
"target": "notes.holdingsNoteTypeId",
"subfield": [
"z"
],
"description": "Holdings note type id",
"applyRulesOnConcatenatedData": true
},
{
"rules": [
{
"conditions": [
{
"type": "trim"
}
]
}
],
"target": "notes.note",
"subfield": [
"z"
],
"description": "Public note data",
"subFieldDelimiter": [
{
"value": " ; ",
"subfields": [
"z"
]
}
],
"applyRulesOnConcatenatedData": true
},
{
"rules": [
{
"value": "false",
"conditions": []
}
],
"target": "notes.staffOnly",
"subfield": [
"z"
],
"description": "If true, determines that the note should not be visible for others than staff",
"applyRulesOnConcatenatedData": true
}
]
}
]
}
MFHD Holdings Statements/Caption Patterns#
FOLIO does not support 85x/86x style pattern/caption holdings statements. To support migration of these patterns to FOLIO, folio_migration_tools provides a holdings statement parsing engine to convert (valid) 85x/86x pairs to textual holdings statements in the FOLIO holdings record, alongside any 866, 867, or 868 textual statements.
Preserving legacy MFHD via Holdings Notes#
One common concern when migrating from systems that manage/exchange holdings data in MFHD format is the loss of some piece of the data that FOLIO is (currently) unable to full represent, especially when MFHDs are not loaded to SRS. To allow continued access to the original data without the need to refer to an offline system, folio_migration_tools offers the ability to map either all MFHD holdings statement fields (853, 854, 855, 863, 864, 865, 866, 867, 868) to a dedicated holdings note in MARC Maker (MRK) format:
=853 \\$81$av.$bno.$i(year)
=863 \\$81.1$a1-16$b1-16$i1994-1998
=866 \\$81.2$av.17no.17 (1999)-v.32no.32 (2003)"
or the entire MFHD record as either MARC Maker (MRK) format or UTF-8 decoded MARC21
=LDR 00330ny 22001453n 4500
=008 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
=090 \\$aKFM4705.6.E3$bA8z
=866 \\$aCOPY 2:
=866 \\$a1982-1984.
=866 \\$a1994-1998.
=866 \\$aCOPY 3:
=866 \\$a1994-1998.
=951 \\$b.b35225907
=951 \\$c.c30246234
=998 \\$sx$c1$r-$v2$l7dc
00330ny a22001453n 4500008004100000090002200041866001200063866001500075866001500090866001200105866001500117951001500132951001500147998002200162\x1e \x1e \x1faKFM4705.6.E3\x1fbA8z\x1e \x1faCOPY 2:\x1e \x1fa1982-1984.\x1e \x1fa1994-1998.\x1e \x1faCOPY 3:\x1e \x1fa1994-1998.\x1e \x1fb.b35225907\x1e \x1fc.c30246234\x1e \x1fsx\x1fc1\x1fr-\x1fv2\x1fl7dc \x1e\x1d
To map only the statement fields, add the following parameter to the MarcHoldingsTransformer task configuration:
{
"includeMrkStatements": true,
"mrkHoldingsNoteType": "Original MARC holdings statements",
}
{
"includeMfhdMrkAsNote": true,
"mfhdMrkNoteType": "Original MFHD Record"
}
{
"includeMfhdMrcAsNote": true,
"mfhdMrcNoteType": "Original MFHD (MARC21)"
}
You can use the name of the holdings note type you want to use, and the tools will resolve the UUID during transformation.
Source=FOLIO-only holdings mapping conditions#
folio_migration_tools support the following custom mapping rule conditions when transforming MARC holdings to source=FOLIO holdings:
set_acquisition_method: This condition will map the LC MFHD standard values from008[7]to their textual representationsset_retention_policy: This condition will map the LC MFHD standard values from008[12]to their textual representationsset_ill_policy: This condition will map the LC MFHD standard values from008[20]to the appropriately-named FOLIO holdings ILL policy reference data valueset_digitization_policy: This condition will map the LC MFHD standard values from008[21]to their textual representationsset_receipt_status: This condition will map the LC MFHD standard values from008[6]to their textual representations (this condition is available for source=MARC records, as well)
Limitations and other considerations#
The MFHD transformer will only act upon the first 852 field encountered. Any remaining 852:s will be reported and discarded.
The MFHD transformer does not handle
852$xor852$zmappings to holdings notes. You will need to create a supplemental MFHD map file or update the FOLIO system default map to include mapping rules for those subfields (or any other subfields of the 852 you want to map to anentity, in FOLIO MARC mapping rules terms) as a “952” mapping rule. During the transformation, the tools will copy the 852 into a 952 field withffindicators and apply the configured 952 mapping.Boundwith information in 014 will not be handled.