Difference between revisions of "Addon:QueryGramplet"

From Gramps
Jump to: navigation, search
(Select and Filter Lists)
(33 intermediate revisions by 4 users not shown)
Line 1: Line 1:
The Query Gramplet takes SQL-like queries and produces a Quick View.
+
{{Third-party plugin}}
 +
{{man warn|Advanced use only|''[[{{Version manual}}_Addons#Manually_installed_Addons|Manually install]] only''. The contributor has marked this preliminary add-on as "partially working with some functionality, but much more work is needed; needs testing"
 +
 
 +
Back up your family tree or use on a copy only.}}
 +
[[File:QueryGramplet-default-51.png|450px|thumb|right|Query Gramplet - default prompt]]
 +
The {{man label|Query Gramplet}} takes SQL-like queries and shows the results in a Quick View.
 +
 
 +
==Usage==
 +
Once [[{{Version manual}}_Addons#Manually_installed_Addons|installed manually]], you can access the {{man label|Query Gramplet}}:
 +
* On the Dashboard by right clicking and selecting {{man label|Query Gramplet}}
 +
* Any of the categories by adding a Gramplet and selecting {{man label|Query}} from the drop down list.
 +
 
 +
Enter your [[QueryGramplet#SQL|SQL-like Query]] and a Quick View result window should be displayed.
 +
 
 +
[[File:QuerySmith.png|thumb|right|400px|Example result]]
 +
{{-}}
  
 
= Examples =
 
= Examples =
  
The QueryGramplet in gramps-addons/trunk for gramps/master can now SELECT, UPDATE, and DELETE. Some examples (keywords are shown
+
The {{man label|Query Gramplet}}Q in Gramps 5.0 can SELECT, UPDATE, and DELETE. Some examples (keywords are shown
capitalized, but the SQL parser is case-insensitive; fields that are capitalized are ''macros'', see below for more information):
+
capitalized, but the SQL parser is case-insensitive; fields that are capitalized are ''macros'' and must be capitalized, see below for more information):
 +
 
 +
<pre>
 +
DELETE FROM person WHERE GIVEN == "Travis";
 +
 
 +
SELECT * FROM person LIMIT 10;
 +
 
 +
SELECT gramps_id, GIVEN, SURNAME FROM person;
 +
 
 +
SELECT event_ref_list[0].ref FROM person;
 +
 
 +
UPDATE person SET GIVEN="Gary" WHERE GIVEN == "Travis";
 +
 
 +
SELECT gramps_id FROM person where ROWNUM < 10;
  
DELETE FROM person WHERE given == "Travis";
+
SELECT gramps_id FROM person LIMIT 5;
  
SELECT * FROM person;
+
SELECT gramps_id FROM person LIMIT 20,30;
  
SELECT gramps_id, GIVEN, SURNAME FROM person;
+
SELECT gramps_id, father_handle.SURNAME, mother_handle.SURNAME from family;  
  
SELECT event_ref_list[0].ref FROM person;
+
UPDATE gramps_id SET tag_list = Tag("Betty") FROM person WHERE "Betty" in primary_name.first_name;
  
UPDATE person SET GIVEN="Gary" WHERE GIVEN == "Travis"
+
</pre>
  
SELECT gramps_id FROM person where ROWNUM < 10;
+
Hints:
  
SELECT gramps_id FROM person LIMIT 5;
+
* You may want to do a general SELECT first ("SELECT * FROM table")--- that will show you the names of fields
 +
* The query will automatically outer-join tables (use FLAT to not join)
 +
* Assigning to a list will append onto it
 +
* Use Tag("name") to lookup or create a new tag
 +
* Use Date(year[, month[, day]]) to create a date
 +
* Use TODAY for a date create for today
 +
* You have access to these libraries/functions: _ (for translations), re, random, db (database)
  
SELECT gramps_id FROM person LIMIT 20,30;
+
Other options:
  
SELECT gramps_id, father_handle.SURNAME, mother_handle.SURNAME from family;
+
* FLAT - do not create extra rows via a JOIN
 +
* EXPAND - do automatic JOINs
 +
* RAW - no extra processing
 +
* NORAW - follow handles, etc
  
 
This API is made possible through the generic struct/json interface. It is very little code, because it relies on these generic structures. It should be able to be made solid enough to expose to users (say as a generic filter). The parser can be made more user friendly... it may just throw an error currently.
 
This API is made possible through the generic struct/json interface. It is very little code, because it relies on these generic structures. It should be able to be made solid enough to expose to users (say as a generic filter). The parser can be made more user friendly... it may just throw an error currently.
Line 31: Line 68:
  
 
Here is the grammar for the subset of SQL supported. The SELECT, UPDATE, DELETE, and LIMIT clauses may be in any order. The WHERE clause (if used) must be last.
 
Here is the grammar for the subset of SQL supported. The SELECT, UPDATE, DELETE, and LIMIT clauses may be in any order. The WHERE clause (if used) must be last.
 +
<pre>
 +
SELECT expr1 [as var1][, expr2 [as var2], ...] FROM table [LIMIT number1[, number2]] [WHERE expression];
  
SELECT expr1 [as var1][, expr2 [as var2], ...] FROM table [LIMIT number1[, number2]] [WHERE expression];
+
UPDATE table SET field1=expr1[, field2=expr2, ...] [LIMIT number1[, number2]] [WHERE expression];
  
UPDATE table SET field1=expr1[, field2=expr2, ...] [LIMIT number1[, number2]] [WHERE expression];
+
DELETE FROM table [LIMIT number1[, number2]] [WHERE expression];
  
DELETE FROM table [LIMIT number1[, number2]] [WHERE expression];
+
... [FLAT | EXPAND] ...
 
 
... [FLAT | EXPAND] ...
 
 
 
... [RAW | NORAW] ...
 
  
 +
... [RAW | NORAW] ...
 +
</pre>
 
'''table''' is one of:
 
'''table''' is one of:
 
+
* [[SQL_Schema#person|person]]
* person
+
* [[SQL_Schema#place|place]]
* place
+
* [[SQL_Schema#repository|repository]]
* repository
+
* [[SQL_Schema#event|event]]
* event
+
* [[SQL_Schema#citation|citation]]
* citation
+
* [[SQL_Schema#source|source]]
* source
+
* [[SQL_Schema#tag|tag]]
* tag
+
* [[SQL_Schema#media|media]]
* media
+
* [[SQL_Schema#family|family]]
* family
 
  
 
Other items:
 
Other items:
Line 85: Line 121:
 
The following are defined for use in your queries:
 
The following are defined for use in your queries:
  
* Tag(name)
+
* <code>Tag(name)</code> - Create or lookup a tag by its name
* re - The Python regular expression library
+
* <code>re</code> - The Python regular expression library
* random - The Python random library
+
* <code>random</code> - The Python random library
* db - the current Gramps database
+
* <code>db</code> - the current Gramps database
* sdb - Simple Database API to the database
+
* <code>sdb</code> - Simple Database API to the database
* Today() - a Gramps Date object set to today's date
+
* <code>Today()</code> - a Gramps Date object set to today's date
* Date() - creates a Gramps Date object
+
* <code>Date()</code> - creates a Gramps Date object
* lib - to access gramps.gen.lib object definitions
+
* <code>lib</code> - to access gramps.gen.lib object definitions
* _(text) - for translations
+
* <code>_(text)</code> - for translations
  
 
Examples:
 
Examples:
 
+
<pre>
SELECT gramps_id, primary_name.surname_list.surname  
+
SELECT gramps_id, primary_name.surname_list.surname  
FROM person  
+
FROM person  
WHERE [re.match("Sm.*th", name) for name in col[1]];
+
WHERE any([re.match("Sm.*th", name) for name in col[1]]);
 
+
</pre>
 
Searches all primary_name surnames to find names that start with "Sm" and end in "th". col[1] is primary_name.surname_list.surname, which is a list of surnames.
 
Searches all primary_name surnames to find names that start with "Sm" and end in "th". col[1] is primary_name.surname_list.surname, which is a list of surnames.
  
Line 229: Line 265:
 
[[Image:QuerySmith.png|thumb|left|400px]]
 
[[Image:QuerySmith.png|thumb|left|400px]]
 
{{-}}
 
{{-}}
 +
<pre>
 +
$ select given_name, surname from people;
 +
 +
$ select * from sources;
  
$ select given_name, surname from people;
+
$ select * from events;
+
 
$ select * from sources;
+
$ select * from families;
 +
</pre>
 +
 
 +
=Issues=
 +
* {{bug|11671}} QueryGramplet: Plugin error (from 'QueryQuickview'): No module named 'gramps.gen.lib.handle'
  
$ select * from events;
+
=See also:=
 +
Original announcement about "Query Gramplet" from developer "Doug Blank"(2013)
 +
* [https://sourceforge.net/p/gramps/mailman/message/31782888/ (Gramps-devel) SQL-like API for Gramps gen.lib], From: Doug Blank - 2013-12-24 16:22:09
 +
** Which mentions feature request {{bug|6513}} ''tool option needed to edit database''
 +
* [http://gramps.1791082.n4.nabble.com/query-py-td1804467.html#a1804468 GRAMPS - Dev - query.py] 2009
  
$ select * from families;
 
  
 +
Because Documentation about data structures becomes quickly outdated, the following cross-references have a parenthetical notation of the vintage - plus the most recent major revision.
 +
* [[Gramps_Data_Model|Gramps Data Model  (v4.1)]] - version 4.1 data diagram (2014)
 +
* [[:Category:GEPS|GEPS]] 032: [[GEPS_032:_Database_Backend_API|Database Backend API]] (proposed 2013, implemented 2015)
 +
* [[:Category:GEPS|GEPS]] 010: [[GEPS_010:_SQL_Backend|SQL Backend/Relational Backend]] (proposed 2008, superseded 2013)
 +
* [[SQLite_Export_Import|SQLite Export Import]] - a [[Third-party_Addons|third party add-on]] which expands the {{Man menu|Family Trees &#x27a1; Import&hellip;/Export&hellip;}} menu options (2009, revised 2019)
 +
* [[Gramps_SQL_Database|Gramps SQL Database]] - documents the format of a SQL database version of the [[Database_Backends#BSDDB_Backend|Gramps BSDDB]] datastore. (2009, revised 2015)
 +
* [[SQL_Schema|SQL Schema]] - (2016)
 +
* [https://www.datacamp.com/community/tutorials/pickle-python-tutorial Pickle Python Tutorial] - '''''not''' specific to Gramps'' (2018)
 
[[Category:Plugins]]
 
[[Category:Plugins]]
 
[[Category:Developers/General]]
 
[[Category:Developers/General]]
 
[[Category:Reports]]
 
[[Category:Reports]]
 
[[Category:Gramplets]]
 
[[Category:Gramplets]]
 +
[[Category:Views]]

Revision as of 23:08, 22 April 2020

Gramps-notes.png

Please use carefully on data that is backed up, and help make it better by reporting any comments or problems to the author, or issues to the bug tracker
Unless otherwise stated on this page, you can download this addon by following these instructions.
Please note that some Addons have prerequisites that need to be installed before they can be used.
This Addon/Plugin system is controlled by the Plugin Manager.

Gnome-important.png
Advanced use only

Manually install only. The contributor has marked this preliminary add-on as "partially working with some functionality, but much more work is needed; needs testing"

Back up your family tree or use on a copy only.

Query Gramplet - default prompt

The Query Gramplet takes SQL-like queries and shows the results in a Quick View.

Usage

Once installed manually, you can access the Query Gramplet:

  • On the Dashboard by right clicking and selecting Query Gramplet
  • Any of the categories by adding a Gramplet and selecting Query from the drop down list.

Enter your SQL-like Query and a Quick View result window should be displayed.

Example result


Examples

The Query GrampletQ in Gramps 5.0 can SELECT, UPDATE, and DELETE. Some examples (keywords are shown capitalized, but the SQL parser is case-insensitive; fields that are capitalized are macros and must be capitalized, see below for more information):

DELETE FROM person WHERE GIVEN == "Travis";

SELECT * FROM person LIMIT 10;

SELECT gramps_id, GIVEN, SURNAME FROM person;

SELECT event_ref_list[0].ref FROM person;

UPDATE person SET GIVEN="Gary" WHERE GIVEN == "Travis";

SELECT gramps_id FROM person where ROWNUM < 10;

SELECT gramps_id FROM person LIMIT 5;

SELECT gramps_id FROM person LIMIT 20,30;

SELECT gramps_id, father_handle.SURNAME, mother_handle.SURNAME from family; 

UPDATE gramps_id SET tag_list = Tag("Betty") FROM person WHERE "Betty" in primary_name.first_name;

Hints:

  • You may want to do a general SELECT first ("SELECT * FROM table")--- that will show you the names of fields
  • The query will automatically outer-join tables (use FLAT to not join)
  • Assigning to a list will append onto it
  • Use Tag("name") to lookup or create a new tag
  • Use Date(year[, month[, day]]) to create a date
  • Use TODAY for a date create for today
  • You have access to these libraries/functions: _ (for translations), re, random, db (database)

Other options:

  • FLAT - do not create extra rows via a JOIN
  • EXPAND - do automatic JOINs
  • RAW - no extra processing
  • NORAW - follow handles, etc

This API is made possible through the generic struct/json interface. It is very little code, because it relies on these generic structures. It should be able to be made solid enough to expose to users (say as a generic filter). The parser can be made more user friendly... it may just throw an error currently.

I'd be interested in any limitations you find, or enhancement ideas.

SQL

Here is the grammar for the subset of SQL supported. The SELECT, UPDATE, DELETE, and LIMIT clauses may be in any order. The WHERE clause (if used) must be last.

SELECT expr1 [as var1][, expr2 [as var2], ...] FROM table [LIMIT number1[, number2]] [WHERE expression];

UPDATE table SET field1=expr1[, field2=expr2, ...] [LIMIT number1[, number2]] [WHERE expression];

DELETE FROM table [LIMIT number1[, number2]] [WHERE expression];

... [FLAT | EXPAND] ...

... [RAW | NORAW] ...

table is one of:

Other items:

  • expr is a field, *, or expression
  • var is an alias
  • number1 by itself is maximum number of rows to select
  • number1 with number2 is start, stop (first row is zero)
  • expression is any valid Python expression

expression and expr may use:

  • random.random() (or other random method)
  • ROWNUM (zero-based counter)
  • col[N] (alias to column)
  • aliases
  • object - the primitive gen.lib object (such as Person, Family, etc)

RAW/NORAW: does not turn the results into strings, but leaves the selected values as raw Python. The default is NORAW. Once set, the new setting will remain the default for this session.

FLAT/EXPAND: if FLAT, then the rows are not cross-product JOINED with other multi-valued columns, but rather left as LISTS. Default is EXPAND. Once set, the new setting will remain the default for this session.

The following shortcuts (also called "macros") can be used in expressions and as a field:

  • SURNAME, short for "primary_name.surname_list[0].surname"
  • GIVEN, short for "primary_name.first_name"

A macro is a low-level text replacement system. We could add other macros, and even allow users to define their own.

Pre-Defined Functions and Libraries

The following are defined for use in your queries:

  • Tag(name) - Create or lookup a tag by its name
  • re - The Python regular expression library
  • random - The Python random library
  • db - the current Gramps database
  • sdb - Simple Database API to the database
  • Today() - a Gramps Date object set to today's date
  • Date() - creates a Gramps Date object
  • lib - to access gramps.gen.lib object definitions
  • _(text) - for translations

Examples:

SELECT gramps_id, primary_name.surname_list.surname 
FROM person 
WHERE any([re.match("Sm.*th", name) for name in col[1]]);

Searches all primary_name surnames to find names that start with "Sm" and end in "th". col[1] is primary_name.surname_list.surname, which is a list of surnames.

UPDATE person SET tag_list=Tag("Smith") WHERE SURNAME == "Smith";

Lists

When a attribute is a list, you can select elements from items in the list, and also filter the list. For example, consider a person's parent_family_list. You can select only a single component, say private, of the parent family like:

SELECT parent_family_list("private") FROM person;

This would select only the private component from the parent families.

Likewise, you can filter the list to, say, only show those families that are private:

SELECT parent_family_list(private=True) FROM person;

This will only show (in the finally selected people) the parent families that are private.

Finally, you can both limit, and select from a list:

SELECT parent_family_list("gramps_id", private=True) FROM person;

That will limit the list to be a list of family gramps_id for private families.

You can delete an entire list by assigning None to it:

UPDATE note_list=None from person;

You can delete an item in a list by assigning None to it:

UPDATE note_list[0]=None from person;

Notes

Some notes on use:

1) Most SQL clauses (UPDATE table, FROM table, SELECT ..., SET field=value, ..., LIMIT ...) can appear in any position, any order

2) ...except the WHERE clause: it must be last; this is because the WHERE clause is not parsed, because:

3) The WHERE clause uses any valid Python expressions. It imports some libraries (such as random), to have ready possible expression needed.

SELECT * from person WHERE random.random() < .1;

This selects records where each has a 10% chance of being selected.

4) The SELECT fields use the bracketed notation for list references. Use "event_ref_list[0]".

5) JOINS are not necessary, because it automatically looks up all relations through the handles. In a SELECT, columns with multiple values in a list will appear as an outer-join with other values in the row.

6) UPDATE will work on any field, through a joined object or on the primary object. For example, you can update the birth date of an event through the person's referenced events.

7) Tables are lowercase, single (not plural) form (eg, person, tag, event).

8) Implemented "LIMIT number", "LIMIT start, stop", and "WHERE ROWNUM < number" (ROWNUM can be used in any expression).

9) Field names are the actual names of the fields of the gramps.gen.lib objects, verbatim, no differences. You might need to look up what you need... no help yet from this interface (although I am working on defining a built-in Schema that could help)

10) .handle or .ref automatically look up their references.

11) Shortcut: you can use col[N] in the WHERE clause to reference a column selected. N is zero-based.

SELECT gramps_id, private FROM person WHERE not col[1];

That will select all people where private is not True. That would include None (non-existent record) and False values. To select only False values, use:

SELECT gramps_id, private FROM person WHERE col[1] == False;

That will not select None values.

12) If an object doesn't match any selected field, it just doesn't show. For example, to find all of the people with at least two surnames on their primary name, use:

SELECT primary_name.surname_list[1] FROM person;

or

SELECT gramps_id, primary_name.surname_list[1] FROM person WHERE col[1];

13) You do not need to reference a field before you can use it in the WHERE clause.

14) The semicolon is optional.

15) Be careful selecting all fields from all records... that could take up a lot of memory, and bring down Gramps.

16) This should be fairly fast, but it does call eval(). This might make things a little slower, but made the code much easier to write. And it does use the full power of python.

17) You can use parentheses in an "UPDATE table SET field=value" value. Something like:

UPDATE table SET field=(field + 1);

but that hasn't been well-tested. (Speaking of testing, there is a Vassilii-inspired unittest with the QueryQuickview... will add more there).

18) The primary_name... stuff is really long and verbose. See "shortcuts" above.

19) Fields that contain other objects, or lists of objects, will show as dictionaries and lists of dictionaries. You can refine those fields by further specifying subparts. Maybe we should not show these, or show in another form...

20) If a selected field does not exist in a record, then it will have a value of None. For example, if you are selecting those people that have a second surname on their primary name, and there are some people who do not have a second surname, it will appear as None. If all columns are None, then the item will not be selected at all.

21) If you know that only one value will match, then a "LIMIT 1" may be a way to speed up the query.

Older documentation for the QueryGramplet in Gramps 3.4

This has different table names (people rather than person), and other names that are different (surname vs primary_name.surname_list[0].surname).

QueryGramplet.jpg


SQL-like:

select FIELDS from TABLE where PYTHON-BOOLEAN-EXPRESSION;

TABLES: FIELDS:

  1. people: given_name, surname, suffix, title, birth_date, death_date, gender, birth_place, death_place, change, marker
  2. families:
  3. sources:
  4. events:

Examples:

$ select * from people where surname.startswith("Smith")
QuerySmith.png


$ select given_name, surname from people;
 
$ select * from sources;

$ select * from events;

$ select * from families;

Issues

  • 11671 QueryGramplet: Plugin error (from 'QueryQuickview'): No module named 'gramps.gen.lib.handle'

See also:

Original announcement about "Query Gramplet" from developer "Doug Blank"(2013)


Because Documentation about data structures becomes quickly outdated, the following cross-references have a parenthetical notation of the vintage - plus the most recent major revision.