Addon:Place completion tool

A tool to bring the places in your GRAMPS database in accordance with the GRAMPS requirements: batch add country, county; look-up latitude-longitude; set description (title); ...

Place Completion tool

This tool helps you fill in the place attributes like county, country, ..., by allowing you to select the places you work on, and do changes on all these places with one button click.

The general aims are:

• Place/Location is a newer concept in GRAMPS. Many older databases only have a Place title field which is a descriptive text containing city, state, country. This should be parsed to insert the values in the correct attribute fields.
• Latitude and longitude are important to show data on a map. However, doing a look-up of this data on the internet is slow and time consuming. The tool allows to search in the free resources on the net.
• Setting of an attribute of a set of places in one go. Eg you give a
• Conversion of latitude and longitude to a fixed data format. On import one might obtain latitude and longitude in several different formats. A conversion tool to store them all in the same format is usefull.
• Construction of a uniform title/description field, from the data in the place object

Design Specification

See Place completion tool specification

Manual

The place completion tool gives a lot of functionality. This manual should help you to understand how it works.

Download resources

The place completion tool can look up for you latitude/longitude, add county information (USA), ... . For some of this functionality, you must download datafiles of the countries you are interested in. Right now you have three options:

1. Download geonames country files. You can do this here freely. Geonames parses fastest, so is the advised format to use
2. Download geonames USA state files. You can do this here freely. This is advised for USA searches, as the data in the USA country file contains many doubles, which can be avoided by searching state per state. State info also contains county information.
3. download GNS Geonet country files (not available for usa). You can do this here freely with ftp.

Watch out, some of these downloads are VERY large, especially USA data. Only download what you need!

Note: The geonames data of popular places is in English, so eg municipalities in Italy will be found, but Roma not, as this is Rome in English. To find data with these you need to search in the localised variants of the name (see below)

DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP

Starting the tool

The placecheck tool is in the tools menu, option tools, under Interactive place completion

The dialog explained

The Dialog consists of 4 parts:

Part 1: selection of places

First you need to choose with which places you want to work. You can use several methods to define your places:

1. Use a place filter. You can use two preset filters: All places, which returns all places, and No Latitude/Longitude given, which returns all places of which the latitude or the longitude is not set. You can also created a custom place filter in the place view, test it with the filter sidebar, and then use it in this tool. All custom filters you made will be available
2. To prevent the need to make a filter for every city, ... in your data, you can set country,state,county,city or parish of the places you want to search on. This works just like in the filter sideview in the places view.
3. Use a latitude, longitude rectangle. Eg, suppose you have the latitude and longitude of all places in the UK, and now want to add in the state attribute Wales, for all places in Wales. You can look on a map, note down the centre of Wales in latitude and longitude, as well as roughly the width and height of this rectangle. This will allow you to obtain all places in Wales (and some in England), allowing to much faster set the state information.

Part 2. Completion of Places

1. The first posibility is to look up in a datafile the latitude and longitude of your places. For this you must have downloaded the necessary resourses, see section above. You can select with a file dialog the file you want to search, and set how this data must be parsed. The following parsing options are available:
   1. GeoNames country file, city search: use the city attribute to look for lat/lon in a GeoNames country file. This is the fastest search.
   2. GeoNames country file, city localized variants search: use the city attribute to look for lat/lon in a GeoNames country file using the localised (non-English) known names in the GeoNames file. Eg, Roma will be found with this option (as Roma is the Italian local variant of the English name Rome)
   3. GeoNames country file, title begin, general search: Use the start of the title field to search in a GeoNames file. With start it is meant everything before a comma:, . This allows to find landmarks, squares, ... . Eg, if the title of your place is: Piazza Navona, Rome, using this search will find you the latitude and longitude of this famous square in Rome.
   4. GeoNames USA state file, city search: Looking for places in the USA file is almost worthless: it takes a long time and every name exists several times. Hence, it is worthwile to use state by state. If a USA state file is selected for doing a search, you must select this option. The city attribute is used for the search.
   5. GNS Geonet country file, city search: use the city attribute to search in a GNS file (slower than GeoNames search!).
   6. GNS Geonet country file, title begin search: use the start of the title of a place to search in a GNS file. With start everything appearing before the first comma is meant.
2. A second option is to parse some existing data in your places.
   1. You can parse the title attribute to extract information from it. Eg a title like Albany, NY can be used to set the city attribute to Albany and the state attribute to NY.
   2. You can set the title of all the selected places to a uniform way. This is interesting if due to imports you have different styles for the title field, which can be annoying in reports. At the moment there are two options:
      1. Set title field to City[, State]: This means the title of your places will contain the city, and if the state field is present, the state will be appended with a comma.
      2. Set title field to Titlestart[, City][, State]: This means the present start of your title will be kept. If this start is not the city, then the city will be appended. If state is present, also state will be appended. An example: suppose your title is Piazza Navona, Italy, the city is Rome and the State is Lazio. Using this option to set the title would change the title attribute into Piazza Navona, Rome, Lazio.
   3. Convert latitude and longitude to a uniform way. Again due to import, copy/paste, you might have latitude and longitude entered in different formats. This is annoying on reports. This options allows you to set for all selected places the lat/lon to one form. The options are:
      1. All in degree notation: use the classical degree notation with degree, minutes and seconds.
      2. All in decimal notation: use the decimal system to denote lat/lon.
      3. Correct -50° in 50°S: a much seen error is to use - for the classical degree notation, which is wrong, and which GRAMPS will not be able to interpret. With this option this error is looked for and corrected.
3. A third option is to set attributes of all selected places. You can set the country, state, county, parish, zip/postal code and city attributes of all places in one sweep.

Part 3. Overview of the results

After having entered all data in Part 1 and 2, you click find for GRAMPS to search all changes that will occur. This part of the dialog shows all changes that will occur.

All selected places are shown. If changes will be done all changes are listed as subentries of the place. Every change will be a subentry.

If the change wil overwrite an existing entry, the subentry is shown in orange.

TO AVOID PROBLEMS, GO OVER ALL CHANGES QUICKLY, AND CHECK ALL ENTRIES IN ORANGE!

The following actions are possible in the result screen:

1. press delete to delete the entry, making sure that this change will not occur. You can delete the entry to delete all changes, or select one subentry, to only delete that specific chagne
2. double-click on an entry to open the place dialog. If you double-click on the entry, all changes will be preentered. If you double-click on a subentry, only this specific change will be preentered in the place dialog.
3. press tab to open in a browser window google maps. Pressing tab on a subentry showing a new lat/lon entry will open google maps on this new lat/lon position. Pressing tab on the top place entry will give open google maps with the old lat/lon position, or if that is not known the title/city field is used for the search.

Part 4. Actions

After you have checked the changes in Part 3, you can now apply them all with one button click, by clicking the Apply button.

Clicking Help will bring you to this page, clicking Close will close the window and clicking Google Maps when an entry is selected in the results field has the same effect as pressing tab on an entry (see above).

Example

Open the example file from the examples where latitude and longitude are empty: example.gramps.

We will now show how the places in this file can be completed.

TO BE COMPLETED. I WILL DO THIS EXAMPLE AS A BUG TEST OF THE CODE, AND THEN MAKE THE CODE AVAILABLE TO ALL.

Troubleshooting

Non UTF-8 latitude/longitude file

The place completion tool expects the input files for location lookup to be in unicode (utf-8). In the occation this is not the case, you will get the error:

File "/home/benny/programms/gramps/gramps2/src/plugins/PlaceCompletion.py", line 851, in load_latlon_file
    self.latlonfile_datastr = infile.read()
  File "/usr/lib/python2.4/codecs.py", line 481, in read
    return self.reader.read(size)
  File "/usr/lib/python2.4/codecs.py", line 293, in read
    newchars, decodedbytes = self.decode(data, self.errors)
 UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1610092-1610094: invalid data 

In the above example it is clear the problem is in two bytes, so you can correct this manually: open the file with eg KHexEdit Binary Editor, go to the specified position (offset 1610092), and change the two bytes with a space.

In the case the file is completely non-unicode, you will have to convert it to unicode with a tool, before using it in the placecompletion tool.

Old Tool

Place Completion Tool

The place completion tool has grown out of a desirement to output places as used in events to maps. For this to work, people must have correct place information in GRAMPS. However, for many users, entering places is neglected: a short description and the city is added, and that's it.

A tool to help you fill in the data needed for places is hence usefull.

The general aims are:

• Place/Location is a newer concept in GRAMPS. Many older databases only have a Place title field with a descriptive text containing city, state, country. This should be parsed to insert the values in the correct attribute fields.
• Latitude and longitude are important to show data on a map. However, doing a lookup of this data on the internet is slow and time consuming. The tool allows to search in the free resources on the net.
• Conversion of latitude and longitude to a fixed data format. On import one might obtain latitude and longitude in several different formats. A conversion tool to store them all in the same format is usefull.
• Construction of a uniform title/description field, from the data in the place object

Design Specification

See Place completion tool specification

Manual

The place completion tool gives a lot of functionality. This manual should help you to understand how it works.

Download resources

The place completion tool can look up for you latitude/longitude, add county information (USA), ... For some of this functionality, you must download datafiles of the countries you are interested in. Right now you have three options:

1. Download geonames country files. You can do this here freely. Geonames parses fastest, so is the advised format to use
2. Download geonames USA state files. You can do this here freely. This is advised for USA searches, as the data in the USA country contains many doubles, which can be avoided by searching state per state. State info also contains county information.
3. download GNS Geonet country files (not available for usa). You can do this here freely with ftp.

Watch out, some of these downloads are VERY large, especially USA data. Only download what you need.

Note: The geonames data of popular places is in English, so eg municipalities in Italy will be found, but Roma not, as this is Rome in English. To find data with these you need to search in the localised variants of the name (see below)

DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP

Starting the tool

The placecheck tool is in the tools menu, option tools, under place completion

This page has good information but needs a cleanup. Please make the Gramps Wiki more useful by re‑organizing this page and linking introductory material. Consider also updating the screenshots.

This is for the present version, the version in development will be different, manual will have to be adapted then. You have to consider 5 actions:

Set filter

Set a place filter which determines which places will be considered. You can set a general filter (standard is All Places). To avoid the need for different filters for every country, state, these can be set in two separate entry fields.

Parse title

Determine if the title must be parsed. Lookup op lat/lon happens with the city field normally. Older grdb/gedcom do not have this fields. Therefore you can parse this field with a regex. This is a comboentry field. Some common options are provided in the dropdown. You can however construct your own regex. Leave this field blank to do no parsing of the title

Look-up latitude/longitude

Search latitude and longitude in data files. A comboentry field is given. You can give your own regex, or you can choose from the dropdown common possibilities. Geonames is fastest. In the file dialog you must select the file you downloaded. The dropdown has the following entries:

• geonames, city search: search populated places with the data in the city search
• geonames, city localised variants : same as before, but now the city is searched in the non-english list. Only well known places (eg Roma, Antwerpen, ...) are listed under the English name (Rome, Antwerp, ...). When English is identical to localized variant, this list is empty and no match can be found with this search
• geonames, title begin general search: searches with what is in the title field before the first ',' in a general fashion: populated places, parks, mountains, ... are returned. Eg: title = Piazza Navona, Rome; will return lat/lon of this famous place in Rome by searching the IT.txt geonames file
• geonames, USA state city search: search in a USA state file, with city. Only populated places are searched
• geonet, city search: search a geonet file, with the city data. Only populated places, and the search is the local variant with accents!
• geonet, title begin geenral search: same as the geonames analog, but on a geonet file

Convert latitude/longitude

Convert lat/lon. Some general conversion on lat/lon are possible, to make sure all data is in the same format. This search also catches mistyped lat/lon and gives a warning. It is possible to correct degree notation with - to correct degree notation.

Set title/description field

Some possibilities to set the title field automatically from the data in the place object are offered.

Do the changes 1: what would change

Now, by clicking find, a list with all places is made, and the changes that can be proposed. These are presented in a tree view with two levels: the general level with placename, and the details with suggested changes. Data overwriting previous data is shown with orange background. You can perform several actions:

• press delete: the entry disappears from the tree view
• press tab : your preferred browser opens a page on google maps. If a lat/lon is suggested, you see this place, otherwise google maps is searched with the place data (city or title)
• select an entry and click on Google Maps: this does the same as pressing tab.
• double-click an entry: the place editor opens, with the data preentered. On the place, all suggested changes are preentered. On a detail, only this specific suggestion is preentered

Do the changes 2: change it

Click Apply: all suggested changes in the treeview are consecutively done. Check for doubles and overwrites before clicking on Apply

The changes cannot be undone. Take a backup if unsure!

DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP

Download

You can download the developers version. You find it at placetool0_9.tar.gz. Put the .glade and .py file both in the plugins directory of GRAMPS 2.2.x.

'DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP