Metagrid enhancer
Metagrid enhancer (beta)
The enhancer helps you to enrich biographical data. It can automate manual searching and copy-pasting of information from third-party providers. It uses the Metagrid network to collect base information and interacts smoothly with other services like Wikidata or GND. Additionally, it provides a simple and unified interface to enrich the project’s biographical data automatically.
The enhancer is still beta. If you find any errors or unexpected behavior, open an issue on GitLab.
Enhancer use case
Historiographical research projects are often scarce on IT-skills. They collect their data in a DB, a wiki, or another data management tool and do their research. Often they manually search third parties for information and copy-paste them into their data management system.
A typical task for a researcher is to identify persons from a source. The researcher then does some research in the Historical Lexicon of Switzerland (hls), Wikidata, or prosopographic databases, and collects basic info about this person. If the person is of interest, the researcher will add this additional info to the database.
The enhancer endpoint can ease this manual process. Researchers can search in Metagrid for the person and link it. Then Metagrid can compile a custom list of properties for this person from different sources and hand it to the researcher. With additional tooling, you can auto-update your dataset. No more copy and paste.
Get the data you need
The enhancer endpoint accepts a configuration as a post-body parameter. In the config, you can define from which source in which priority you want to gather data. The following example will return a field last_name with values from the Historical Lexicon of Switzerland (hls) or, if the hls has no data, from Wikidata.
{
"fields": {
"last_name": {
"sources": {
"1": {
"field": "LastName",
"target": "local",
"provider": "hls-dhs-dss"
},
"2": {
"field": "P734",
"target": "external",
"provider": "wikidata"
}
}
}
}
}
You can post the config to the enhancer endpoint and get the following result.
curl 'https://api.metagrid.ch/enhance/dodis/person/5.json' -X POST --data-raw '{"fields":{"last_name":{"sources":{"1":{"field":"LastName","target":"local","provider":"hls-dhs-dss"},"2":{"field":"P734","target":"external","provider":"wikidata"}}}}}'
{
"fields": {
"last_name": {
"reference": "https://hls-dhs-dss.ch/de/articles/004647/2014-04-22/",
"values": "Petitpierre"
}
},
"warning": "Currently, the API is under heavy development, so changes might occur."
}
You get an object of fields that contains all the requested resources. Each field has a values and a reference. The values contains the result from the requested source. In our example, this is the string “Petitpierre”, but this could also be an object. The reference points back to the source of the value. The origin of the value “Petitpierre” is https://hls-dhs-dss.ch/de/articles/004647/2014-04-22/
Transform data
Federated fetching is very handy, but nothing new. It simplifies a complex problem and can make your life easier. Unfortunately, you often get data that does not fit your schema/identifiers e.g. the birthplace “Bern” has the id=25 in your data management tool, but the hls references it with the id=120003. To some extent, the enhancer can transform between different schemas or identifiers. To use this advanced feature, the data of your project have to be in Metagrid and/or the geolinker. In the following example, we query the family relations of Gottlieb Emanuel Haller (hallernet) from the gnd:
curl 'https://api.metagrid.ch/enhance/hallernet/person/00407.json' -X --data-raw '{"fields":{"family":{"sources":{"1":{"field":"familialRelationship","target":"external","provider":"gnd","transformer":"metagrid"}}}}}'
{
"fields": {
"family": {
"reference": "https://d-nb.info/gnd/119500019",
"values": [{
"@id": "https://d-nb.info/gnd/1254552383",
"preferredName": "Anna Margaretha von Haller",
"relationship": "Ehefrau",
"same_as": [{
"provider": "gnd",
"@id": "https://d-nb.info/gnd/1254552383",
"identifier": "1254552383"
}, {
"provider": "burgerbibliothek",
"@id": "https://archives-quickaccess.ch/bbb/person/209084",
"identifier": "209084"
}, {
"provider": "viaf",
"@id": "http://viaf.org/viaf/924164913181918980009",
"identifier": "924164913181918980009"
}, {
"provider": "hallernet",
"@id": "https://hallernet.org/data/person/00417",
"identifier": "00417"
}]
}, {
"@id": "https://d-nb.info/gnd/118545140",
"preferredName": "Albrecht von Haller",
"relationship": "Vater",
"same_as": [{
"provider": "histoirerurale",
"@id": "https://www.histoirerurale.ch/pers/personnes/4289.html",
"identifier": "4289"
}, {
"provider": "sudoc",
"@id": "http://www.idref.fr/028090462/id",
"identifier": "028090462"
}, {
"provider": "hls-dhs-dss",
"@id": "https://hls-dhs-dss.ch/de/articles/010656/2009-11-05/",
"identifier": "010656"
}, {
"provider": "hallernet",
"@id": "https://hallernet.org/data/person/01200",
"identifier": "01200"
}, {
"provider": "histhub",
"@id": "https://data.histhub.ch/person/11415365",
"identifier": "11415365"
}, {
"provider": "hfls",
"@id": "http://www.hfls.ch/humo-gen/family/1/F20065?main_person=I59982",
"identifier": "F20065/I59982"
}, {
"provider": "gnd",
"@id": "https://d-nb.info/gnd/118545140",
"identifier": "118545140"
}, {
"provider": "burgerbibliothek",
"@id": "https://archives-quickaccess.ch/bbb/person/19405",
"identifier": "19405"
}, {
"provider": "viaf",
"@id": "http://viaf.org/viaf/49234879",
"identifier": "49234879"
}, {
"provider": "loc",
"@id": "http://id.loc.gov/authorities/names/n50018444",
"identifier": "n50018444"
}, {
"provider": "helveticat",
"@id": "https://www.helveticat.ch/discovery/search?query=lds50,contains,118545140\u0026vid=41SNL_51_INST:helveticat",
"identifier": "118545140"
}, {
"provider": "bsg",
"@id": "https://www.bsg.nb.admin.ch/discovery/search?query=lds50,contains,118545140\u0026vid=41SNL_54_INST:bsg",
"identifier": "118545140"
}, {
"provider": "kbga",
"@id": "https://kbga.karl-barth.ch/actors/2867",
"identifier": "2867"
}]
}, {
"@id": "https://d-nb.info/gnd/140325409",
"preferredName": "Hans Kaspar Schulthess",
"relationship": "Schwiegervater",
"same_as": [{
"provider": "histhub",
"@id": "https://data.histhub.ch/person/11523716",
"identifier": "11523716"
}, {
"provider": "gnd",
"@id": "https://d-nb.info/gnd/140325409",
"identifier": "140325409"
}, {
"provider": "hallernet",
"@id": "https://hallernet.org/data/person/00928",
"identifier": "00928"
}, {
"provider": "hfls",
"@id": "http://www.hfls.ch/humo-gen/family/1/F41483?main_person=I122994",
"identifier": "F41483/I122994"
}, {
"provider": "hls-dhs-dss",
"@id": "https://hls-dhs-dss.ch/de/articles/028467/2010-07-01/",
"identifier": "028467"
}, {
"provider": "viaf",
"@id": "http://viaf.org/viaf/103853477",
"identifier": "103853477"
}]
}, {
"@id": "https://d-nb.info/gnd/118701061",
"preferredName": "Carl Ludwig von Haller",
"relationship": "Sohn",
"same_as": [{
"provider": "hfls",
"@id": "http://www.hfls.ch/humo-gen/family/1/F20364?main_person=I60825",
"identifier": "F20364/I60825"
}, {
"provider": "histhub",
"@id": "https://data.histhub.ch/person/11415374",
"identifier": "11415374"
}, {
"provider": "hls-dhs-dss",
"@id": "https://hls-dhs-dss.ch/de/articles/010657/",
"identifier": "010657"
}, {
"provider": "viaf",
"@id": "http://viaf.org/viaf/95145949",
"identifier": "95145949"
}, {
"provider": "helveticat",
"@id": "https://www.helveticat.ch/discovery/search?query=lds50,contains,118701061\u0026vid=41SNL_51_INST:helveticat",
"identifier": "118701061"
}, {
"provider": "bsg",
"@id": "https://www.bsg.nb.admin.ch/discovery/search?query=lds50,contains,118701061\u0026vid=41SNL_54_INST:bsg",
"identifier": "118701061"
}, {
"provider": "sudoc",
"@id": "http://www.idref.fr/034071474/id",
"identifier": "034071474"
}, {
"provider": "gnd",
"@id": "https://d-nb.info/gnd/118701061",
"identifier": "118701061"
}, {
"provider": "burgerbibliothek",
"@id": "https://archives-quickaccess.ch/bbb/person/19460",
"identifier": "19460"
}, {
"provider": "hallernet",
"@id": "https://hallernet.org/data/person/03258",
"identifier": "03258"
}, {
"provider": "loc",
"@id": "http://id.loc.gov/authorities/names/n84105734",
"identifier": "n84105734"
}]
}, {
"@id": "https://d-nb.info/gnd/13987349X",
"preferredName": "Karl von Haller",
"relationship": "Enkel",
"same_as": [{
"provider": "hfls",
"@id": "http://www.hfls.ch/humo-gen/family/1/F25140?main_person=I74042",
"identifier": "F25140/I74042"
}, {
"provider": "viaf",
"@id": "http://viaf.org/viaf/298760523",
"identifier": "298760523"
}, {
"provider": "histhub",
"@id": "https://data.histhub.ch/person/11433734",
"identifier": "11433734"
}, {
"provider": "burgerbibliothek",
"@id": "https://archives-quickaccess.ch/bbb/person/19440",
"identifier": "19440"
}, {
"provider": "gnd",
"@id": "https://d-nb.info/gnd/13987349X",
"identifier": "13987349X"
}, {
"provider": "hls-dhs-dss",
"@id": "https://hls-dhs-dss.ch/de/articles/013432/2011-03-03/",
"identifier": "013432"
}, {
"provider": "helveticat",
"@id": "https://www.helveticat.ch/discovery/search?query=lds50,contains,13987349X\u0026vid=41SNL_51_INST:helveticat",
"identifier": "13987349X"
}]
}]
}
},
"warning": "Currently, the API is under heavy development, so changes might occur."
}
The transformer tries to convert the gnd identifier into other identifiers. This works if the related person is in Metagrid. The result of such a transformation is fuzzy and just delivers data, if Metagrid is aware of all the involved parties. Very often you will get no result – but some data is better than no data.
Build your enhancer schema
Generating a schema is challenging. You need a lot of knowledge about each of our partners. To make this easier we provide a Gui and a playground, to build and test your schema.
Open the playground and start from scratch
Give the field a name and select a provider. For most providers, the target is local. This means the enhancer makes a lookup in the Metagrid DB. For the provider wikidata and gnd you can select the target external. In this case, the enhancer reaches out to the provider and asks for a value. This opens the possibility to get data not present in Metagrid.
On top of this definition, you can add a transformer. A transformer tries to translate the resulting data into additional representations of the same data. A transformer adds a same_as key to the result, in which you can find additional identifiers for the referencing resource. For example, with the Metagrid transformer, you can get all the identifiers from Metagrid for a given resource. This can help to map the result to your database.
Try your enhancer schema
After you build your schema it’s time to test it. Select a provider and an identifier and query the enhancer. You will get the result at the bottom of the page.
Typescript client
To integrate the enhancer into your workflow, we provide a basic typescript client. Just generate a schema with the schema gui and enhance a single or all resources for your provider. With a callback function, you can do anything with the data you need.
// The enhancer schema
const options: EnhancerOptions = {
schema: '{"fields":{"last_name":{"sources":{"1":{"provider":"hls-dhs-dss","target":"local","field":"LastName"}},"default":""}}}'
};
// instantiate a new enhancer with given options
const enhancer = new Enhancer(options);
// enhance a single person identified by provider and identifier
enhancer.get("provider", "identifier", async function (data: EnhancerResponse): Promise<void> {})
// enhance all persons from provider
enhancer.all("provider", async function (data: EnhancerResponse): Promise<void> {})
A more advanced example is available in the example folder of the project. This is a real-world use case to enhance some fields in Metagrid.
