View source for GEPS 045: Place Model Enhancements/tr

Rethinking Gramps Place data model

One of the items on the Gramps [[5.2 Roadmap]] is to look into better GEDCOM support for places.  We could continue to force fit GEDCOM import data to Gramps, or look into expanding Gramps data model to make for a better fit.
== Place Enhancements ==
=== Place.Type: ===
Gramps only supports a single place type.
Gov (http://gov.genealogy.net/search/index) and GEDCOM L group support multiple types, each with a subordinate date range, GEDCOM L group support a citation for each type as well
==== Recommendation ====
Make place types a list.  Each element should have a subordinate date range and will have a citation list. (done)

We should also provide a routine similar to the current place displayer to return a place type for a particular date. (done, gen.utils.location.__get_type)

=== Place Types: ===
Changes to place types, do not actually modify the data model, but dealing with them better will certainly impact a lot of code.

GeoNames currently doesn’t do much to provide place types.  It currently uses PCLx, for Country, PPLx for 'Populated Place', and ADM1-ADMn for different level of administrative place.  These do not map well to Gramps PlaceTypes.  The Place Cleanup Gramplet tries to do it anyway, but allows user modification.

Gov currently has over 260 place types in several broad categories.  The categories are Administrative, Civil, Religious (church), Judicial, populated place, transportation, and a few miscellaneous others.  They are still adding more.  The GetGov Gramplet currently stores the returned German place type as a Custom Gramps place type, which is of course not translatable.  For example, Deutsches Reich, or USA gets a PlaceType of 'Staat', rather than country.  While the system has a lot of types it is very German centric, many types have only German names in their definition documents although some of the more commonly used types have at least English translations and some have several other translations.
==== Recommendation ====
One of;
#) Adopt the Gov type system.  We would have to do a lot of translation work on many types, or indicate that the type is local to certain countries (so could use the local words).  Our GUI would probably have to use at least a two level hierarchy in the PlaceType selector, possibly three levels, to help users find appropriate types for manually entered places or filtering.
#: One of the levels (top level?) would be the category type.
#: An additional level might be the country specific types.
#) Provide a conversion of the Gov type to the closest Gramps PlaceType, possibly adding a few more Gramps PlaceTypes.  If we did that, I think it would be reasonable to store the original Gov type in a place fact.
#) Keep our standard Place Types, allow GOV types to be added as custom types as needed.
#: Add a concept of Place type groups, which make it possible for multiple types to be identified as Country (for example).  The Groups are G_COUNTRY, G_REGION, G_PLACE, G_UNPOP, G_BUILDING.
#: The GUI type selector is enhanced to support additional levels corresponding to the groups (only when needed).
#: The Edit/Preferences/Place Types (new panel) is added to allow editing of the place type names, addition and removal of place types, and assignment to groups.  It is also possible to add and remove groups.


The third option is accepted for development. (done)

=== Place.Name: ===
Gramps supports multiple names, each with a language and date range.

GEDCOM supports two additional parts of a name; neither of which seem to be used much and are not currently supported by Gramps.

FONE (Place phonetic variation) and its subordinate TYPE.  This seems to be used for far eastern places for example if hiragana was used to provide a reading of a name written in kanji, then the <PHONETIC_TYPE> value would indicate kana. The TYPE is defined as [<user defined> | hangul | kana]

ROMN (PLACE_ROMANIZED_VARIATION:= {Size=1:120}) and its subordinate TYPE.  The romanized variation of the place name is written in the same form prescribed for the place name used in the superior <PLACE_NAME> context. The method used to romanize the name is indicated by the line_value of the subordinate <ROMANIZED_TYPE>, for example if romaji was used to provide a reading of a place name written in kanji, then the <ROMANIZED_TYPE> subordinate to the ROMN tag would indicate ‘romaji’.  The TYPE is defined as [<user defined> | pinyin | romaji | wadegiles]

GEDCOM L group adds in _NAMC (place name addition) for each name.  Their documentation currently says "Addition to PLACE_NAME, can be appended to PLACE_NAME in non-delimited reports to make the label unique".  This is for suffixes like "am Main" in "Frankfurt am Main".  

GEDCOM L group adds in ABBR with a subordinate TYPE.  ABBREVIATION_OF_NAME Abbreviation for the place name, whereby the type of abbreviation can be further explained with the optional TYPE.  

GEDCOM L group adds in a citation list for each name.

The GeoNames data and Place Cleanup Gramplet currently store place name abbreviations as an alternate name with the 'abbr' language code.

The GeoNames data and Place Cleanup Gramplet currently store colloquial place names ('Big Apple' for New York City USA) as an alternate name with no language code.
==== Recommendation ====
#) Our primary name and alternate names become part of a more general place name list.  Each has a date, language, and a citation list associated with it.
#) Add an abbreviation list to each place name.  Each abbreviation has a type and value.  In Reports and graphs, when abbreviations were specified to be used, the language/date specific name abbreviation would be used instead by the place displayer.
#) For place name abbreviations, the place display format editor allows selection of the abbreviation by type of first available in the list for a particular name.  If no abbreviation is present, use a place displayer routine uses the full name.

Done

=== Postal Code: ===
Gramps currently offers a single field 'Code' to store postal codes.  It may have also been used for other purposes in the past.

GeoNames, GOV, GEDCOM L all support multiple postal codes.  GEDCOM L adds in a subordinate DATE range and citation for each code.

The Place Cleanup Gramplet currently stores multiple postal codes in the code field separated by commas.
==== Recommendation ====
One of the following:
#) Make the postal code(s) part of a more general place attribute list.  Each code (attribute) would have a citation list associated with it.  Since an attribute doesn't have a date, the date would be appended as text to the attribute value (Ex: "12345; Date: JAN 12 2019").  (Done)

=== MAIDENHEAD_LOCATOR ===
GEDCOM L has added this to the Place import/export.

The Maidenhead Locator divides the world into small areas (micro-fields) marked by 8 letters and numbers.  It was originally conceived for use in ham radio, but it is also used as a geographic locator in some other applications.  https://en.wikipedia.org/wiki/Maidenhead_Locator_System
==== Recommendation ====
Store the Maidenhead locator in the place attribute list.  Date and Citation parts would not be used.  (Done)

=== ADMINISTRATIVE_IDENTIFIER ===
GEDCOM L has added this to the Place import/export.

The official or public identifier for a location object, such as community code, ISO 3166 for countries and states.  GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.
==== Recommendation ====
Store the ADMINISTRATIVE IDENTIFIER in the place attribute list. Each would have a citation list associated with it. Since an attribute doesn't have a subtype, the TYPE_OF_ADMINISTRATIVE_IDENTIFIER would be appended as text to the attribute value (Ex: "BlahBlah; Type: whatever").  (Done)

=== DEMOGRAPHIC DATA ===
GEDCOM L has added this to the Place import/export.
Demographics about people to the place object, such as number of households, number of people, occupations, and so on.  GEDCOM L allows multiple of these, each with date range and an additional TYPE and citation.
==== Recommendation ====
Store the DEMOGRAPICAL_DATA in the place attribute list. Each would have a citation list associated with it. Since an attribute doesn't have a subtype or date, the TYPE_OF_DEMOGRAPICAL_DATA and/or date would be appended as text to the attribute value (Ex: "43210; Type: Population; Date: 2017"). (Done)

=== EVENT_DETAIL ===
GEDCOM L has added this to the Place import/export.

GEDCOM L has added an event reference to the place.  
==== Recommendation ====
Add an Event reference list to the Place.  The list could contain one of the following:
#) EventRef;  The Event ref role would be set to 'Place' (Done)
#) Direct pointer to Event object.  (Rejected)

== Enclosed by ==
The following items would be part of the Gramps place reference, which currently only includes a date and the reference to another place.
=== HIERARCHICAL_RELATIONSHIP ===
GEDCOM L has added this to the Place import/export.

Values are [POLI | RELI | GEOG | CULT] to differentiate political (administrative), ecclesiastical, geographical or cultural attributions.  GEDCOM L includes this to provide some indication of the type of enclosure.  For example a Church might be enclosed by a city and enclosed by a parish, with different relationships.
=== Citation ===
GEDCOM L adds a citation to the Enclosed by
==== Recommendation ====
* The Place Type Enclosure (PlaceRef) is extended with a PlaceHierType.  It can be one of ADMIN, RELI, GEOG, CULT or JUDI (or other custom types).  The PlaceRef also has an added citation list.

== Place Attributes ==
To deal with several new facts associated with places, a Place Attributes list is added.  Each place attribute includes the standard type, value, notes and citations.

Facts that also include a date or subtype are encoded as text in the Attribute.value.  Several new attribute types are added to support places AIDN (Administrative Identifier), DMGD (Demographic data), MAIDEN (Maidenhead Locator), and POSTAL (Postal code).

The Attribute Type selector in the GUI has a second level menu called place with these types in it.

== Citations ==
GEDCOM L has added citations to most of its additional data.

The Place object has citation lists attached to Places, Place Attributes, Place Names, Place Types, and Place Enclosed by.

== ToDo ==
=== Top priority ===
If we do these first and carefully, we should be able to run Gramps with new trees.

gen.lib.place
* &#x2714; update object, get_schema, serialize/unserialize etc.
* &#x2714; Add add_eventref, get/set/merge_eventreflist
* &#x2714; Add add/get/set/merge place attributes
* &#x2714; get/set/merge_place_types (get/set effects only the top of the list of types). New add_place_type, Supports sortin the most modern place type (based on date) with admin hierarchy at top of list (makes views.placemodel a bit faster to pick type to view and allows for get_place_type to return top value for compatibility).
* <s>Remove get/set title</s>
* &#x2714; Get/set_name – Place.name removed, one unified list of names, so these methods only effect the top of a list of names.   Also remove get/set alternative names, and added add_name (which can sort the most recent name to top of list).
* &#x2714; Add get/set/merge_abbreviation
* &#x2714; Remove get/set_code and Place.code (on import/upgrade this is moved to Place Attribute)
* <s>Remove get/set/merge_alternative_locations</s>
* &#x2714; fix up other methods as needed.
* &#x2714; add citationrefs to place names, types, placerefs
<s>gen.lib.placefact – new.  Similar to gen.lib.attribute but with date.</s>

<s>gen.lib.placefacttype – new. Similar to lib.eventtype, but for place facts.</s>

&#x2714; gen.lib.placetype – update for place groups and support three classifications of types; standard Gramps types, numbered custom types (supports GOV types), and full custom types.  Also update place type combo menu to allow much larger type list in two level menu.

&#x2714; gui.views.placemodel – update for changed serialization.  Remove 'Code' also change plugins.lib.libplaceview for missing column.

&#x2714; gen.lib.placeabbrev, gen.lib.placeabbrevtype, gen.lib.placehiertype

=== 2nd priority ===
==== Editors ====
I'm assuming that the place editor is not significantly changed in style from the current one.
* &#x2714; Add place type editor icon to place type.  Place Type combo remains but only affects top of list.
* &#x2714; Remove 'Code' from Place editor
* &#x2714; Add place Types tab gui.editors.displaytabs.placetypeembeddedlist and placetypemodel
* <s>Remove Title edit capability</s>
* &#x2714; Update Place Names tab for unified list of names, abbreviations
* &#x2714; Add Place Attributes tab gui.editors.displaytabs.placeatttrembeddedlist
* &#x2714; Add place Events tab - gui.editors.displaytabs.placeeventembeddedlist
* &#x2714; Duplicate place editor work for gui.editors.editplaceref
* &#x2714; New Place type editor gui.editors.editplacetype. Combo for type and Add date and a citationref tab
* &#x2714; gui.editors.editevent – fix DeleteEventQuery to scan for places containing events
* &#x2714; Update Place Names Editor for unified list of names.
* &#x2714; Add abbreviations and citations tabs to place name editor gui.editors.displaytabs.placeabbrevembedlist and placeabbrevmodel
* &#x2714; New Place Abbreviations Editor gui.editors.editplaceabbrev
* &#x2714; Add citations tab to gui.editors.editplaceref

==== Views ====
* &#x2714; Add Place Attributes and Events Gramplets to Place views plugins.lib.libplaceview
* &#x2714; Modify Events Gramplet plugins.gramplets.events to deal with place events (remove age column).
* &#x2714; update Attributes Gramplet plugins.gramplets.attributes for places
* &#x2714; plugins.lib.maps.placeselection update to use place groups instead of types
* &#x2714; plugins.lib.maps.geography update for place changes
* &#x2714; plugins.view.eventview fix DeleteEventQuery to support place events
* &#x2714; plugins.view.geoplaces modify marker color to use much larger place types
* &#x2714; plugins.view.placetreeview fix add place to add heierarchy type

==== Import/export/upgrade ====
* &#x2714; Update dtd
* &#x2714; Update XML export
* &#x2714; Update XML import
* &#x2714; Modify bsddb update routines plugins.db.bsddb.upgrade
* &#x2714; Create dbapi update routine plugins.db.pbapi.upgrade, and hooks into db loaders
DB upgrade or XML import should:
* &#x2714; Move Code field into place attribute
* &#x2714; add new citations lists
* &#x2714; change place type to type list with date and citations
* &#x2714; change place name and place alternative names to place name list with new abbreviations
* &#x2714; change PlaceRef to add citations and hierarchy
* &#x2714; add placetype custom information to metadata

==== Merge ====
* &#x2714; Fix gen.merge.mergeeventquery to check for place events
* &#x2714; Remove Code from gui.merge.mergeplace and its glade file

==== Place Displayer ====
* &#x2714; Update place displayer gen.display.place to support rules for place types, groups, and abbreviations etc.
* &#x2714; Update gen.utils.location to support date for multiple place types, hierarchy, and to return abbreviations
* &#x2714; rewrite gui.editors.editplaceformat and its glade file

These now support:
* multiple place formats (like before)
* each format supports a heirarchy type and language, and allows display order reversal
* each format can have multiple rules, which can be added, removed and reordered (order is executed top to bottom)
* each rule can apply to all places, or just one place (useful for country specific rules)
* each rule can apply to a place Group, Type, or the street/number
* For Group rules, you can select the Group from the standard and custom groups
* for Type rules, you can select the Type from the standard and custom types
* for the Group or Type rules, you can choose to display All, largest, or smallest of matching types, or hide them.
* for the Group or Type rules, you can choose to show an abbreviation from the abbreviation types, or the first available abbreviation type, or None (which shows the associated name, instead of the abbreviation).
* for street/number rules you can choose to display street/number, number/street, or Hidden
* The Place Formats are stored in the db NOT in a system wide file, so only applies to the current tree.  This is due to potential references to specific places or customized types or groups which may not be present in other trees.

==== Edit/Preferences/Place Types ====
* &#x2714; Updated gui.configure with new Place Types panel

This now supports:
* Shows the place type list on left side, along with used indication, and a comment
* Shows place groups for selected Place Type on right side
* An Entry at top right allows the Place Type name to be changed
* The Groups are shown as check boxes, which can be changed
* There is a search box at bottom of dialog
* You can add new types via the Add button at bottom
* You can remove types that are not used and are not standard
* You can hide types (from type selection combobox menus) with the Hide button.
* You can add new Groups via a right-click over one of the current Group checkbox items
* You can remove Groups via a right-click over one of the current Group checkbox items
* The customized place type/group information is stored in the current db and so applies to only a specific tree.
* The customized place type/group information is stored in the XML for backup.

=== 3rd priority ===
==== Import/Export ====
* &#x2714; Update Gedcom import/export from GEPS 043 work to support place model https://github.com/gramps-project/gramps/pull/469
* &#x2714; Update importer for place changes; csv
* &#x2714; Update importer for place changes;  geneweb
* &#x2714; Update importer for place changes; progen
* (N/A) Update importer for place changes;vcard
* &#x2714; Update exporter for place changes; csv
* (N/A) Update exporter for place changes;ftree
* (N/A) Update exporter for place changes;geneweb
* (N/A) Update exporter for place changes;vcalendar
* (N/A) Update exporter for place changes;vcard

==== Reports ====
* &#x2714; Plugins.webreport.narrativeweb, etc.  Add Place Formats option, Update to use place groups, show place attributes.
* &#x2714; Plugins.graph.gvfamilylines – update to use place Displayer/Formats.
* &#x2714; Plugins.graph.gvrelgraph – Uses_place displayer, update to use place Displayer/Formats.
* &#x2714; Plugins.libllibstubkeyword – updated to support place groups for Country, State (largest REGION), County (smallest Region), City (largest Place).  Also fixed up postal to find in place attributes. Title now uses standard Place Displayer with preferred format, Place sub-codes 1-5 can use the first five Place displayer Formats.
* &#x2714; Plugins.drawreport.statisticschart – Uses_place displayer, update to use place Displayer/Formats.
* &#x2714; Plugins.textreport.ancestorreport – Uses_place displayer, update to use place Displayer/Formats.
* &#x2714; Plugins.textreport.placereport – added more names and types to the report, fixed to avoid problems with Place Events.
* Plugins.mapservices.eniroswedenmap  uses some specific place types…

==== Tools ====
* Generate Testcases  may want to add some additional place items
* Check & Repair; check_events needs to look at places for references.
==== Filters ====
* &#x2714; gen.proxy.filter - support place attributes and event refs for sanitizing notes
* &#x2714; gen.proxy.private - support place names, types and their citation lists as well as the new eventrefs and attributes.
* &#x2714; gen.proxy.referencedbyselection - support place names, types and their citation lists as well as the new eventrefs and attributes.
* &#x2714; gen.proxy.proxybase - added new get_placehier_types, get_placeabbr_types, get_place_attribute_types methods. 
* &#x2714; gen.filters.rules.place.has_data - remove "code" place type scan should look at whole list
* &#x2714; New filter rules for place attributes, place events (gen.filters.rules.place._hasattribute, _hasevent)
* &#x2714; gui.filters.sidebar._placessidebarfilter - Place Sidebar filter; remove code

==== Addons ====
* CheckPlaceTitles – don't need this anymore?
* &#x2714; D3Charts/DescendantIndentedTree - has get_title for a place
* DataEntryGramplet – has set_title for a place
* &#x2714; DescendantBookReport – has get_title for a place
* &#x2714; DetailedDescendantBookReport– has get_title for a place
* &#x2714; DescendantsLines\substkw – stop override for PlaceFormat from Plugins.libllibstubkeyword, so just use the plugins.lib version.
* &#x2714; DynamicWeb – uses place.get_title place get_main_location, get_alternate_locations, get_code (done, works, but may want to enhance with more display of place multiple types, attributes, events...).
* &#x2714; ExtractCity – uses set_code
* &#x2714; GetGov – modify to store data in correct parts of place, adapt to updated Place Types with hierarchy and date.    
* &#x2714; GraphView – add support for place selectable formats
* &#x2714; ImportMerge – make sure algorithm doesn't break on Event/Place/Event loop
* &#x2714; lxml - update to support new schema
* &#x2714; PlaceCleanup – Need to store data in correct parts of place, change Code handling, change abbreviation handling, store attributess, multiple place types etc.
* &#x2714; PlaceCompletion – don't support this anymore?
* &#x2714; Sqlite.ExportSql – update for new place data (Not compatible with previous sql files)
* &#x2714; Sqlite.ImportSql – update for new place data (Not compatible with previous sql files)

==== Belgeler ====
* [[Gramps_Data_Model|Veri modeli şemasını]] güncelle
* Viki'yi güncelleyin (veri girme, yer adı düzenleyici, …)

==== Test code ====
* Db tests
* &#x2714; Import/export/report tests
* Filter tests

== Comments ==
== See also ==
*[[GEPS 006: Better Place handling]]
*[[GEPS 043: Improving GEDCOM support for Places]]
*[[GEPS 045: Place Model Enhancements - Place Changes Screenshots]]

[[Category:GEPS|P]]