Difference between revisions of "Tips for large databases"

From Gramps
Jump to: navigation, search
(conversion of TipsForLargeDatabases)
 
(Avoid Gramplets)
 
(37 intermediate revisions by 10 users not shown)
Line 1: Line 1:
Large data files, what to do, and what not
+
{{man tip|The advice on this page was for older versions of Gramps so may not work for you. Please update as needed.}}
[[Category:Documentation]]
+
{{stub}}<!--check accuracy  / Incorporate "Import tested with the GRANDMA Mennonite database of 1.4 million people. by user on reddit!! https://www.reddit.com/r/gramps/comments/dzevcl/database_size_limit_for_gramps/fb6hdbj/ " -->
  
 +
Large family tree data files, what to do, and what not to do.
  
 
==Loading the file==
 
==Loading the file==
 +
{{man note|Should work for BSDDB versions of Gramps}}
 +
Initial import of a large (100,000+) database from either Gramps XML formats or GEDCOM can take a few hours. You will need to adjust the number of allowable locks. For 140,000 people you should use:
 +
* ''max_locks'' 300000
 +
* ''max_objects'' 300000
  
 +
The easiest way to do this is to:
 +
# create a new Family Tree
 +
# exit Gramps
 +
# add a '''<code>DB_CONFIG</code>''' file to the database directory before importing (see '''<code>gramps -l</code>''' output for the directory of a specific family tree.)
  
For small files, you can work on GEDCOM or GRAMPS XML format. This works great.
+
Contents of this DB_CONFIG file should be:
However, these formats are completely in you computer memory, so for larger files, they become slow, and might bring your computer to a standstill.
+
#may want to fiddle with cachesize also
 
+
#set_cachesize 0 200000000 2
Therefore, you should only handle large files by the GRAMPS GRDB format. You do this by creating a new GRDB file, and importing the large file into it. This can take a long time! Do it in the evenings. If you enable transactions in menu Edit->Preferences, the importing of data will be much faster
+
set_lk_max_locks 300000
 +
set_lk_max_objects 300000
  
 +
{{-}}
  
 
== Default view: bookmarks and active person ==
 
== Default view: bookmarks and active person ==
  
 +
As there are many people, loading the person view might take some time. See a comparison of [[Gramps Performance|performance]] on large datasets, between different Gramps versions.
  
As there are many people, loading the person view might take some time. See the performace results: [[GRAMPS Performance]].
+
You can circumvent this load time by going to the [[Gramps Relationships Category view screenshot|relationships view]] before opening the family tree. This will open the active person and their family extremely fast, independent of the amount of data you gathered.
  
 +
You can add bookmarks to common people, or in the branch you are researching. This allows to change person in the relationships view without the need to activate the person in the person view first.
  
You can circumvent this load time, by going to the relationships view before opening the grdb file. This will open the active person and his family extremely fast, independent of the amount of data you gathered.
+
== Avoid relationship calculation ==
  
 +
Avoid displaying relationships on the status bar. From the {{man menu|Edit > Preferences}} ''Display'' tab change the '''Status bar''' option from {{man label|Relationship to the home person}} to {{man label|Active person's name and ID}}.
  
You can add bookmarks to common people, or in the branch you are researching. This allows to change person in the relationships view without the need to activate the person in the person view first.
+
== Avoid Gramplets ==
 +
Avoid gramplets which do a lot of database work. The '''[[Deep_Connections_Gramplet|Deep Connections]]''' Gramplet seems to be the worst case. Those type of gramplets slow everything down enormously.
  
 +
== Avoid general filters ==
  
== Avoid general filters ==
+
Filters search all the data looking for the people who satisfy your query. Try to avoid this on large files.
  
 +
{{man tip|Search letters|Fast search can be done by enabling column search (disable {{man label|Sidebar}} on {{man menu|View}} menu), but this will search all letters on words into the column, not only the first letters.}}
  
Filters go over all the data looking for the people who satisfy your query. Try to avoid this on large files.
+
For a surname search:
 +
* click  on a person in the [[Gramps People Category screenshots|people view]]. Now start typing the family name of the person you need. The view will automatically go to the family name as you complete your typing.
 +
* for a more detailed search, use the keyboard shortcut {{man key press|CTRL|F}}. This will expand all the nodes and allows to search more specifically.
  
For name searches do the following:
+
== Flat views are faster than Tree views ==
* click in the person view on a person. Now start typing the family name of the person you need. The view will automatically go to the family name
 
* for more detailed search, click ''CTRL-F'' first. This will expand all nodes and allows to search more specifically.
 
  
 +
Using a large database and person treeview needs to often rebuild the view.
  
== Please contribute ==
+
Flat person view (list) is faster.
  
If you more tips for large databases, please share them
+
[[Category:Documentation]]

Latest revision as of 03:27, 20 October 2020

Tango-Dialog-information.png
The advice on this page was for older versions of Gramps so may not work for you. Please update as needed.

Gramps-notes.png

Please update or expand this section.


Large family tree data files, what to do, and what not to do.

Loading the file

Gramps-notes.png
Should work for BSDDB versions of Gramps

Initial import of a large (100,000+) database from either Gramps XML formats or GEDCOM can take a few hours. You will need to adjust the number of allowable locks. For 140,000 people you should use:

  • max_locks 300000
  • max_objects 300000

The easiest way to do this is to:

  1. create a new Family Tree
  2. exit Gramps
  3. add a DB_CONFIG file to the database directory before importing (see gramps -l output for the directory of a specific family tree.)

Contents of this DB_CONFIG file should be:

#may want to fiddle with cachesize also
#set_cachesize 0 200000000 2
set_lk_max_locks 300000
set_lk_max_objects 300000


Default view: bookmarks and active person

As there are many people, loading the person view might take some time. See a comparison of performance on large datasets, between different Gramps versions.

You can circumvent this load time by going to the relationships view before opening the family tree. This will open the active person and their family extremely fast, independent of the amount of data you gathered.

You can add bookmarks to common people, or in the branch you are researching. This allows to change person in the relationships view without the need to activate the person in the person view first.

Avoid relationship calculation

Avoid displaying relationships on the status bar. From the Edit > Preferences Display tab change the Status bar option from Relationship to the home person to Active person's name and ID.

Avoid Gramplets

Avoid gramplets which do a lot of database work. The Deep Connections Gramplet seems to be the worst case. Those type of gramplets slow everything down enormously.

Avoid general filters

Filters search all the data looking for the people who satisfy your query. Try to avoid this on large files.

Tango-Dialog-information.png
Search letters

Fast search can be done by enabling column search (disable Sidebar on View menu), but this will search all letters on words into the column, not only the first letters.


For a surname search:

  • click on a person in the people view. Now start typing the family name of the person you need. The view will automatically go to the family name as you complete your typing.
  • for a more detailed search, use the keyboard shortcut CTRL+F. This will expand all the nodes and allows to search more specifically.

Flat views are faster than Tree views

Using a large database and person treeview needs to often rebuild the view.

Flat person view (list) is faster.