# MARC Rules based mapping ## Introduction ## The mapping rules FOLIO stores its MARC-to-FOLIO mapping rules as a JSON object that you can download and edit via: ``` /mapping-rules/marc-{bib,holdings} ``` These endpoints supports `GET` and `PUT` requests. You can find documentation for the mapping options in the `mod-source-record-manager` [GitHub repository](https://github.com/folio-org/mod-source-record-manager/blob/master/RuleProcessorApi.md). ## Authority Transformation >Authority transformation in FOLIO migration tools is deprecated. Please use FOLIO's Data Import app to load authorities in FOLIO. ## Bib Transformation As of version 1.9.0 of `folio_migration_tools`, we rely on a hybrid approach to migrating MARC-based bibliographic data into FOLIO that puts a lot of the heavy lifting of mapping from MARC-to-Instance on FOLIO's data import system. Historically, we have maintained our own implementation of FOLIO's MARC-mapping system in Python as part of these tools, but this functionality has never had full feature parity with FOLIO's rules processing system. Development in this area will be low-priority moving forward. By default this tool will perform a basic mapping, including Title, Primary Contributor, and some format information to create a set of instance records with `source=FOLIO` and a file of MARC21 records that can be loaded via Data Import using the default single-record import import profile (or a copy), to convert them to `source=MARC` and perform a full MARC-to-Instance map. This process can be automated via a BatchPoster record type (coming soon) or using [`folio_data_import`](https://github.com/FOLIO-FSE/folio_data_import)'s `folio-marc-import` script. ### Configuring Your MARC Bib Transformer Task The first step in preparing to transform MARC Bib records for FOLIO is to create a task configuration: ```json { "name": "bibs", "migrationTaskType": "BibsTransformer", "ilsFlavor": "tag001", "hridHandling": "preserve001", "updateHridSettings": false, "files": [ { "file_name": "bibs_suppressed.mrc", "discoverySuppressed": true }, { "file_name": "bibs.mrc", "discoverySuppressed": false } ] } ``` #### Excluding some MARC records from the output file for Data Import You may wish to exclude MARC records for a subset of your migrated bibs from the Data Import file produced by BibsTransformer (eg. lendable equipment records, or other materials that do not require MARC source records). To accomplish this, you can set `data_import_marc: false` in the file configuration for the specific MARC file containing those records. For example: ```json { "name": "bibs", "migrationTaskType": "BibsTransformer", "ilsFlavor": "tag001", "hridHandling": "preserve001", "updateHridSettings": false, "files": [ { "file_name": "bibs_suppressed.mrc", "discoverySuppressed": true }, { "file_name": "bibs.mrc", "discoverySuppressed": false }, { "file_name": "equipment.mrc", "discoverySuppressed": false, "data_import_marc": false } ] } ``` MARC records from `equipment.mrc` will not be saved to the `folio_marc_instances_bibs.mrc` file produced by this task, but `source=FOLIO` instances will be. (posting-bibtransformer-marc-records)= ### Posting BibTransformer MARC records While `folio_migration_tools`'s `BatchPoster` task does not currently support posting the MARC21 records generated by BibsTransformer, the file can be loaded manually, via the Data Import app in FOLIO, or by using the `folio-marc-import` script available as part of the `folio_data_import` module: Example: ```shell folio-marc-import --gateway_url "https://folio-snapshot-okapi.dev.folio.org" --tenant_id diku --username diku_admin --password admin --marc_file_path "iterations/test/results/folio_marc_instances_bibs.mrc" --split-files --split-size 10000 --batch_size 100 ``` ## MFHD Transformation MFHD records are a great way to export records from your legacy system to migrate to FOLIO. However, FOLIO's support for storing MFHD records in SRS is limited. If you are using these tools to migrate MFHD-based holdings records, we recommend creating `source=FOLIO` holdings records and _not_ loading the corresponding MARC data to FOLIO ### Configure Your MFHD Transformer Task The first step in preparing to transform MARC Bib records for FOLIO is to create a task configuration: ```json { "name": "transform_mfhd", "migrationTaskType": "HoldingsMarcTransformer", "legacyIdMarcPath": "001", "locationMapFileName": "location_map.tsv", "defaultCallNumberTypeName": "Library of Congress classification", "fallbackHoldingsTypeId": "82747568-cdf3-4980-bba2-e5b38950f65b", "holdingsTypeUuidForBoundwiths": "1b6c62cf-034c-4972-ac80-fa595a9bfbde", "hridHandling": "default", "createSourceRecords": false, "boundwithRelationshipFilePath": "bib_mfhd.tsv", "supplementalMfhdMappingRulesFile": "custom-marc-holdings.json", "files": [ { "file_name": "mfhd_suppressed.mrc", "discoverySuppressed": true }, { "file_name": "mfhd.mrc", "discoverySuppressed": false } ], "updateHridSettings": false }, ``` For more information on the task configuration options for MFHD transformation, see [HoldingsMarcTransformer](./tasks/holdings_marc_transformer) ### Boundwith Mapping for MFHD Holdings ("Voyager-style" boundwiths) The MFHD transformer supports migrating boundwith relationships — where multiple bibliographic records share a single physical holdings record — via the `"boundwithRelationshipFilePath"` configuration key. The boundwith relationship file is a TSV placed in `source_data/holdings/` with columns `MFHD_ID` and `BIB_ID` mapping each holdings record to one or more bib records: ```text MFHD_ID BIB_ID 12345 100001 12345 100002 12346 100003 ``` During transformation, the tool creates a copy of the holdings record for each additional bib (instance), setting the `holdingsTypeId` to the value of `"holdingsTypeUuidForBoundwiths"` and generating deterministic UUIDs for the copies. The resulting relationship map (`boundwith_relationships_map.json`) is written to the results folder and consumed by the [ItemsTransformer](./tasks/items_transformer) to create `boundwithPart` records linking items to their boundwith holdings. #### Configuration example ```json { "name": "transform_mfhd", "migrationTaskType": "HoldingsMarcTransformer", "legacyIdMarcPath": "001", "locationMapFileName": "location_map.tsv", "defaultCallNumberTypeName": "Library of Congress classification", "fallbackHoldingsTypeId": "82747568-cdf3-4980-bba2-e5b38950f65b", "holdingsTypeUuidForBoundwiths": "1b6c62cf-034c-4972-ac80-fa595a9bfbde", "boundwithRelationshipFilePath": "bib_mfhd.tsv", "hridHandling": "default", "createSourceRecords": false, "files": [ {"file_name": "mfhd.mrc"} ] } ``` (supplemental-mfhd-mapping-rules)= ### Supplemental MFHD Mapping Rules As of v1.9.0, `folio_migration_tools` supports the use of a supplemental MFHD mapping rules configuration, in lieu of either updating the system rules or creating a full, customized copy of the rules (specified by `mfhdMappingFileName` in the task configuration). Instead, you can create a JSON file with an object containing only the rules you want to add (or modify). The tools will perform a python `dict.update()` of the full rules file using the file specified in `"supplementalMfhdMappingRulesFile"`. For example, a supplemental mapping to map public and private holdings notes from 852$x and 852$z (using rules for a 952$x and 952$z, see [Limitation and Other Considerations](#limitations-and-other-considerations) below): ```json { "952": [ { "entity": [ { "rules": [ { "conditions": [ { "type": "set_holdings_note_type_id", "parameter": { "name": "Note" } } ] } ], "target": "notes.holdingsNoteTypeId", "subfield": [ "x" ], "description": "Holdings note type id", "applyRulesOnConcatenatedData": true }, { "rules": [ { "conditions": [ { "type": "trim" } ] } ], "target": "notes.note", "subfield": [ "x" ], "description": "Holdings private note", "subFieldDelimiter": [ { "value": " ; ", "subfields": [ "x" ] } ], "applyRulesOnConcatenatedData": true }, { "rules": [ { "value": "true", "conditions": [] } ], "target": "notes.staffOnly", "subfield": [ "x" ], "description": "If true, determines that the note should not be visible for others than staff", "applyRulesOnConcatenatedData": true } ] }, { "entity": [ { "rules": [ { "conditions": [ { "type": "set_holdings_note_type_id", "parameter": { "name": "Note" } } ] } ], "target": "notes.holdingsNoteTypeId", "subfield": [ "z" ], "description": "Holdings note type id", "applyRulesOnConcatenatedData": true }, { "rules": [ { "conditions": [ { "type": "trim" } ] } ], "target": "notes.note", "subfield": [ "z" ], "description": "Public note data", "subFieldDelimiter": [ { "value": " ; ", "subfields": [ "z" ] } ], "applyRulesOnConcatenatedData": true }, { "rules": [ { "value": "false", "conditions": [] } ], "target": "notes.staffOnly", "subfield": [ "z" ], "description": "If true, determines that the note should not be visible for others than staff", "applyRulesOnConcatenatedData": true } ] } ] } ``` ### MFHD Holdings Statements/Caption Patterns FOLIO does not support 85x/86x style pattern/caption holdings statements. To support migration of these patterns to FOLIO, `folio_migration_tools` provides a holdings statement parsing engine to convert (valid) 85x/86x pairs to textual holdings statements in the FOLIO holdings record, alongside any 866, 867, or 868 textual statements. ### Preserving legacy MFHD via Holdings Notes One common concern when migrating from systems that manage/exchange holdings data in MFHD format is the loss of some piece of the data that FOLIO is (currently) unable to full represent, especially when MFHDs are not loaded to SRS. To allow continued access to the original data without the need to refer to an offline system, `folio_migration_tools` offers the ability to map either all MFHD holdings statement fields (853, 854, 855, 863, 864, 865, 866, 867, 868) to a dedicated holdings note in MARC Maker (MRK) format: ```text =853 \\$81$av.$bno.$i(year) =863 \\$81.1$a1-16$b1-16$i1994-1998 =866 \\$81.2$av.17no.17 (1999)-v.32no.32 (2003)" ``` or the entire MFHD record as either MARC Maker (MRK) format or UTF-8 decoded MARC21 ```text Maker =LDR 00330ny 22001453n 4500 =008 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ =090 \\$aKFM4705.6.E3$bA8z =866 \\$aCOPY 2: =866 \\$a1982-1984. =866 \\$a1994-1998. =866 \\$aCOPY 3: =866 \\$a1994-1998. =951 \\$b.b35225907 =951 \\$c.c30246234 =998 \\$sx$c1$r-$v2$l7dc ``` ```text 00330ny a22001453n 4500008004100000090002200041866001200063866001500075866001500090866001200105866001500117951001500132951001500147998002200162\x1e \x1e \x1faKFM4705.6.E3\x1fbA8z\x1e \x1faCOPY 2:\x1e \x1fa1982-1984.\x1e \x1fa1994-1998.\x1e \x1faCOPY 3:\x1e \x1fa1994-1998.\x1e \x1fb.b35225907\x1e \x1fc.c30246234\x1e \x1fsx\x1fc1\x1fr-\x1fv2\x1fl7dc \x1e\x1d ``` To map only the statement fields, add the following parameter to the `MarcHoldingsTransformer` task configuration: ```json { "includeMrkStatements": true, "mrkHoldingsNoteType": "Original MARC holdings statements", } ``` ```json { "includeMfhdMrkAsNote": true, "mfhdMrkNoteType": "Original MFHD Record" } ``` ```json { "includeMfhdMrcAsNote": true, "mfhdMrcNoteType": "Original MFHD (MARC21)" } ``` You can use the name of the holdings note type you want to use, and the tools will resolve the UUID during transformation. ### Source=FOLIO-only holdings mapping conditions `folio_migration_tools` support the following custom mapping rule conditions when transforming MARC holdings to source=FOLIO holdings: * `set_acquisition_method`: This condition will map the LC MFHD standard values from `008[7]` to their textual representations * `set_retention_policy`: This condition will map the LC MFHD standard values from `008[12]` to their textual representations * `set_ill_policy`: This condition will map the LC MFHD standard values from `008[20]` to the appropriately-named FOLIO holdings ILL policy reference data value * `set_digitization_policy`: This condition will map the LC MFHD standard values from `008[21]` to their textual representations * `set_receipt_status`: This condition will map the LC MFHD standard values from `008[6]` to their textual representations (this condition is available for source=MARC records, as well) ### Limitations and other considerations * The MFHD transformer will only act upon the first 852 field encountered. Any remaining 852:s will be reported and discarded. * The MFHD transformer does not handle `852$x` or `852$z` mappings to holdings notes. You will need to create a supplemental MFHD map file or update the FOLIO system default map to include mapping rules for those subfields (or any other subfields of the 852 you want to map to an `entity`, in FOLIO MARC mapping rules terms) as a "952" mapping rule. During the transformation, the tools will copy the 852 into a 952 field with `ff` indicators and apply the configured 952 mapping. * Boundwith information in 014 will not be handled.