A tool to bring the places in your GRAMPS database in accordance with the GRAMPS requirements: batch add country, county; look-up latitude-longitude; set description (title); ...
This tool '''only''' works with version 2.2.5+ of GRAMPS! [[Place_completion_tool#Download|Download]].
[[Category:Proposed Tool Specifications]]
*Construction of a uniform title/description field, from the data in the place object
== Manual ==
== Design Specification ==
See [[Place completion tool specification]]
Clicking ''Help'' will bring you to this page, clicking ''Close'' will close the window and clicking ''Google Maps'' when an entry is selected in the results field has the same effect as pressing tab on an entry (see above).
=== Example ===
Open the example file from the examples where latitude and longitude are empty: [http://gramps.svn.sourceforge.net/viewvc/*checkout*/gramps/trunk/gramps2/example/gramps/example.gramps?revision=7186 example.gramps].
We will now show how the places in this file can be completed.The best thing to do is open a new family tree (.grdb), give it a name, and import the example.gramps file. This file has 852 places, which would mean a lot of manual edits if you do not use this tool! Now, open the place view. You will see all places are of the form::Aberdeen, WAThis value is the <code>Place Name</code> attribute (the title or description of the place). === Step 1: City and State data ===Our first step will be to split this field into a <code>City</code> value (here Aberdeen), and a <code>State</code> value (here WA). We open the ''Place completion tool'':[[Image:place_completion_tool_example1.png|500px|Parse the Place Name Field]]Here we have selected ''All Places'', and we parse the title as ''City [,|.] State''. Click on ''Find'', quickly scan the data if all looks ok, and then click on ''Apply''. You are notified that 851 places are updated. This is one less that the number of places. Indeed, one place does have a different type of title: ''Puerto Rico'' has no state information. === Step 2: Look-up latitude and longitude ===We have downloaded the GeoNames datafiles for the USA states, and will now use that to complete the latitude and longitude of the data. At the same time, this will fill up the county field. [[Image:place_completion_tool_example2.png|500px|Look up lat/lon for Alaska]] In the above screenshot, you see we have selected All Places with State=AK. In the second part of the window we give that we want to search in the AK_DECI.txt file downloaded from GeoNames, using the parsing method: ''GeoNames USA state file, city search''. Note that if you want to change AK into Alaska, this would be possble. Just set state=Alaska in the set attributes section of the window.
TO BE COMPLETEDDo this now for all the states. I WILL DO THIS EXAMPLE AS A BUG TEST OF THE CODEAlways check for doubles. Eg, for state ''AL'', going over the changes, we encounter:[[Image:place_completion_tool_example3.png|Double in lat/lon, AND THEN MAKE THE CODE AVAILABLE TO ALLcity Enterprise exists in two counties]]We see that the first time 'Enterprise' if found, it is in county ''Coffee'' in lat/lon:31.31/-85.85. The second hit is for county ''Chilton'' with lat/lon:32.73/-86.62.
== Troubleshooting ===== Non UTF-8 latitudeYou can now use the Google Maps button (or press TAB key) while the lat/longitude file ===The place completion tool expects the input files for location lookup lon subentry is selected to be see where this city is in unicode (utf-8)both cases. In the occation From this it will be clear for example that one is a hamlet, not the casereally a city, you will get while the error: <code><nowiki>File "/home/benny/programms/gramps/gramps2/src/plugins/PlaceCompletionfirst is a real city.py"So now, line 851, in load_latlon_file self.latlonfile_datastr = infile.read() File "select the second lat/usr/lib/python2.4/codecs.py"lon entry, line 481, in read return selfand delete it by pressing the DEL key.readerDo the same for the second county entry.read(size) File "/usr/lib/python2.4/codecs.py", line 293, in read newcharsIn case google maps did not allow you to determine which is the correct city, decodedbytes = self.decodeyou can double click on the city to open the Place Dialog ('''Warning: this will preenter the data, selfof the Place Completion tool.errors) UnicodeDecodeError: So hit cancel here if you want to exit without these changes done'utf8' codec can't decode bytes in position 1610092-1610094: invalid data</nowiki></code> ). In this dialog the references tab allows you to navigate to all events coupled to this place. This will give you extra information you might use to decide which of the two found places is the correct place.
In the above example it is clear the problem is === Step 3: Problem entries===While updating all places in two bytesstep 2, so you can correct this manuallywill have noticed some errors in the state information: Some places have a dubious state: open the file with eg <code>KHexEdit Binary Editor</code>, go to the specified position (offset 1610092), and change the two bytes with a space. OH-AL
In You can obtain these states by choosing ''All Places'' en setting the case the file is completely nonstate search box to '''-unicode, '''. Clicking Find will give you will have all these problem places. You can use google maps or the place dialog to convert it sort them out. You can also use the USA country GeoNames file to unicode with a tool, before using it search these places in the placecompletion toolentire USA. You will need sufficient memory for this, or you will obtain a MemoryError (see below)!
=== Step 4: Lat/Lon not found ===
= Old Tool === After the above, still some 45 places have no latitude/longitude found. You can now select these places by setting the Place Completion Tool==filter to 'No Latitude/Longitude', which will find you all places with no coordinates.
The place completion tool has grown out It will be clear that many of a desirement to output places as used in events to mapsthose can be quickly corrected: abbreviations, eg the city field contains ''St. For this to workGeorge'', which should be ''Saint George''; double names, people must have correct place information in GRAMPS. Howevereg Waterloo-Cedar Falls, for many usersIA means Waterloo near Cedar Falls, entering places is neglected: a short description and changing the city is added, to Waterloo and that's itredoing the search using Google Maps will allow to quickly find which coordinates for Waterloo are needed.
A tool to help you fill in the data needed == Advanced Usage ==This is for places is hence usefulladvanced users only knowing regular expressions.
The general aims are:*Place/Location is a newer concept in GRAMPS. Many older databases only parsing fields have a Place title field with a descriptive text containing city, state, country. This should be parsed entry fields allowing you to insert the values in the correct attribute fieldsgive your own parsing.*Latitude and longitude are important to show data on a mapParsing uses regular expressions. However, doing a lookup of You can use this data on the internet is slow and time consuming. The tool allows to search in the free resources on the net.*Conversion of latitude parse your title, and longitude to parse a fixed data format. On import one might obtain latitude and longitude lat/lon file in several different formatsyour own way. A conversion tool to store them all in For reference, here an overview of the parsing codes used for the same format is usefull.predefined parses:*Construction of a uniform === Parse title/description fielddetails ===The following regex expressions are used, from the data in the place objectwhere for brevity we use some variables defined lower.
== Design Specification ==#<nowiki>"City [,|.] State" is parsed by : r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+state_translated +r'>.+?)\s*$'</nowiki>See #<nowiki>"City [,|.] Country" is parsed by : r'\s*(?P<'+city_translated +r'>.+?)\s*[Place completion tool specification.,]\s*(?P<'+country_translated +r'>.+?)\s*$'</nowiki>#<nowiki>"City (Country)" is parsed by : r'\s*(?P<'+city_translated +r'>.*?)\s*\(\s*(?P<'+country_translated +r'>[^\)]+)\s*\)\s*$'</nowiki>#<nowiki>"City" is parsed by : r'\s*(?P<'+city_translated +r'>.*?)\s*$'</nowiki>
Here the variables used are: lat_translated =_('lat')lon_translated = Manual _('lon')city_translated =_('city')county_translated =_('county')The place completion tool gives a lot of functionality. This manual should help you to understand how it works.state_translated = _('state')country_translated = _('country')
=== Download resources ===The place completion tool You can look up for you latitude/longitudeuse one of these variables as a group, add county information (USA)and the tool will recognise them, ... For some of this functionality, you must download datafiles of and use as values for the countries you are interested incorresponding place attributes. Right now you have three options:
#Download geonames country files. You can do this [http:=== Lat/Lon lookup parsing ===For the regex of lat/download.geonames.org/export/dump/ here freely]. Geonames parses fastestlon lookup, so is the advised format you need to use#Download geonames USA state files. You can do this [http://geonames.usgs.gov/domestic/download_data.htm here freely]. This is advised indicate which data must be replaced with existing place attributes for USA searchesthe search, as the data in the USA country contains many doubles, well as indicate which can regex groups must be avoided by searching state per state. State info also contains county information.#download GNS Geonet country files (not available for usa). You can do this [ftp://ftp.nga.mil/pub/gns_data here freely with ftp]extracted.
Watch out#<nowiki>"GeoNames country file, some of these downloads are city search" is parsed with: r'\t'+CITY_transl +r'VERY\t[^\t]*\t[^\t]*\t'+latgr + r'[\d+-][^\t]*)\t' large+ longr + r'[\d+-][^\t]*)\tP'</nowiki>#<nowiki>"GeoNames country file, city localized variants search" is parsed with: r'[\t,]'+CITY_transl+r'[,\t][^\t\d]*\t?' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\tP'</nowiki>#<nowiki>"GeoNames country file, especially title begin, general search" is parsed with: r'\t'+TITLEBEGIN_transl +r'\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\t[PSTV]'</nowiki>#<nowiki>"GeoNames USA data. Only download what you need.state file, city search" is parsed with: r'\t'+CITY_transl+r'\tPopulated Place\t[^\t]*\t[^\t]*\t' + countygr + r'[^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)'</nowiki>#<nowiki>"GNS Geonet country file, city search" is parsed with: r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\tP\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+CITY_transl+r'\t[^\t]*\t[^\t\n]+$'</nowiki>#<nowiki>"GNS Geonet country file, title begin search" is parsed with: r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[PLSTV]\t[^\t]*\t[^\t]*'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+TITLEBEGIN_transl+r'\t[^\t]*\t[^\t\n]+$'</nowiki>
<small>'''Note''': The geonames For extraction of data of popular places is you can use the same groupnames as in Englishtitle parsing, so eg municipalities latgr in Italy will be found, but Roma not, as this is Rome in English. To find data with these you need to search in the localised variants of the name above should read: <nowiki>r'(see below)?P<'+lat_translated +r'>'</smallnowiki>.
The syntax for the values that need to be used for searching in the file, eg CITY_transl, is given by : _('CITY'). You can use as substitution values:_('DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUPCITY'), _('TITLE'), _('TITLEBEGIN'), _('STATE'), _('PARISH').
=== Starting the tool ===The placecheck tool is will read in the given regex, replace the substitution strings by the values in the tools menuplace object, option toolsdo the search, under ''place completion''and extract the regex groups given from the result.
{{cleanup}}''This is for the present version, the version in development will be different, manual will have to be adapted then.''You have to consider 5 actionsResource: [http:====Set filter====Set a place filter which determines which places will be considered//de. You can set a general filter (standard is All Places). To avoid the need for different filters for every country, state, these can be set in two separate entry fieldswikipedia.org/wiki/Wikipedia:GEOnet_Names_Server GEOnet]
==Troubleshooting ====Parse title=Non UTF-8 latitude/longitude file ===Determine if The place completion tool expects the title must input files for location lookup to be parsedin unicode (utf-8). In the occation this is not the case, you will get the error: <code><nowiki>File "/home/benny/programms/gramps/gramps2/src/plugins/PlaceCompletion.py", line 851, in load_latlon_file self. Lookup op latlatlonfile_datastr = infile.read() File "/usr/lib/lon happens with the city field normallypython2. Older grdb4/gedcom do not have this fieldscodecs.py", line 481, in read return self.reader. Therefore you can parse this field with a regexread(size) File "/usr/lib/python2. This is a comboentry field4/codecs. Some common options are provided py", line 293, in the dropdownread newchars, decodedbytes = self.decode(data, self. You errors) UnicodeDecodeError: 'utf8' codec can however construct your own regex. Leave this field blank to do no parsing of the title't decode bytes in position 1610092-1610094: invalid data</nowiki></code>
====Look-up latitude/longitude====Search latitude Note that the Place Completion tool catches this errors and longitude in data filesshows you an information box. A comboentry field is given. You can give your own regexAfter this, or you can choose from the dropdown common possibilities. Geonames is fastest. In the file dialog you must select tool will attempt to read the file you downloaded. The dropdown has the following entries:*geonames, city search: search populated places with the data in the city search*geonames, city localised variants : same as before, but now the city is searched in the nonutf-english list. Only well known places 8 (eg Roma, Antwerpen, ...unicode) are listed under the English name (Rome, Antwerp, ignoring errors...). When English is identical to localized variantThis might give good results, this list is empty and no match can be found with this search*geonames, title begin general search: searches with what is in the title field before the first ',' in a general fashion: populated places, parks, mountains, ... are returned. Eg: title = Piazza Navona, Rome; but will return lat/lon of this famous place in Rome by searching the IT.txt geonames file*geonames, USA state city search: search in a USA state file, with city. Only populated places are searched*geonet, city search: search a geonet file, with the city data. Only populated places, and the search is the local variant with accents!*geonet, title begin geenral search: same as the geonames analog, but off course fail to produce results on a geonet filenon-unicode encoded files
====Convert latitude/longitude====Convert latIn the above example it is clear the problem is in two bytes, so you can correct this manually: open the file with eg <code>KHexEdit Binary Editor</lon. Some general conversion on lat/lon are possiblecode>, go to make sure all data is in the same format. This search also catches mistyped lat/lon specified position (offset 1610092), and gives change the two bytes with a warning. It is possible to correct degree notation with - to correct degree notationspace.
====Set title/description field====Some possibilities to set In the title field automatically from case the data file is completely non-unicode, you will have to convert it to unicode with a tool, before using it in the place object are offeredplacecompletion tool.
====Do the changes 1: what would change =Memory Error ===Now, by clicking find, a list The tool might fail with all places is made, and the changes that can be proposed. These are presented in a tree view with two levelserror: the general level with placename, and the details with suggested changes. Data overwriting previous data is shown with orange background. You can perform several actions:*press delete: the entry disappears from the tree view*press tab : your preferred browser opens a page on google maps. If a lat/lon is suggested, you see this place, otherwise google maps is searched with the place data (city or title)*select an entry and click on Google Maps: this does the same as pressing tab.*double-click an entry: the place editor opens, with the data preentered. On the place, all suggested changes are preentered. On a detail, only this specific suggestion is preentered
<nowiki>self.latlonfile_datastr ====Do the changes 2: change it ====infile.read() File "/usr/lib/python2.4/codecs.py", line 481, in read return self.reader.read(size)Click Apply: all suggested changes File "/usr/lib/python2.4/codecs.py", line 293, in the treeview are consecutively doneread newchars, decodedbytes = self. '''Check for doubles and overwrites before clicking on Apply''' decode(data, self.errors) MemoryError</nowiki>
The changes cannot tool has to load the datafile for latitude/longitude searching into memory. For large files like USA.txt, this might be undoneimpossible if you have limited memory. You can try to close as many programs running at together with GRAMPS, and try the tool again. Take a backup if unsure!
'''DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP'''
== Download ==
You can download the developers version. You find it at [http://cage.ugent.be/~bm/varia/placetool0_9placecompletion_1_0.tar.gz placetool0_9placecompletion_1_0.tar.gz]. Put the .glade and .py file both in the plugins directory of GRAMPS 2.2.x6+.
'''DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP''
