Building Gramps AIO cx freeze-based

Gramps 4.2.8 and later. Any comments and help (e.g. on wiki pages) are welcome.

The Gramps "software bundle" for Windows or GrampsAIO is an all-in-one (a.k.a. "AIO") installation package of Gramps which includes all dependencies for the Windows platform.

This tutorial describes how to generate a Windows AIO package from a development installation such as is created with Gramps for Windows with MSYS2.

It is intended for Windows users and does not attempt to describe how to do the same thing on Linux or a Mac.

("AIO" means all-in-one, which means that everything you need to run Gramps is included in the one AIO file you download and then install -- all the related programs and library files which Gramps needs on Windows.)

Note The first Gramps AIO package was created by an advanced Windows power-user named Josip. He is the one who evolved the AIO package and modified the AIO package over the years as Gramps has changed. Everything documented here initially done by him; this just documents some changes and tries to explain it for others. Any Gramps Windows user owes him a large debt of gratitude.

Note In this tutorial you will be creating a 32-bit AIO. The steps for creating a 64-bit version are quite similar, just a few editing changes to the files and scripts.

Prerequisites

You must have a development version of Gramps up and running. See Gramps for Windows with MSYS2 which describes the process for a 64-bit installation. When following this process, you must change a few things for 32-bits.

Note Pay close attention to the shell to use when running the various commands. Use MSYS2 MSYS for the pacman and makepkg-mingw steps and the MSYS2 MINGW32/64 shell to run python3 Gramps.py.

When the instructions say to use the MSYS2 MINGW64 shell you should use MSYS2 MINGW32 shell instead for 32-bit operation.

Change the 'pacman' command install file prefixes from mingw-w64-x86_64 to mingw-w64-i686.

Because Josip does not have (as of this writing) some of the packages (gtk, db, bsddb3) that he built done for 32-bit we have to do them ourselves 'the hard way'.

Change the MINGW_INSTALLS=mingw64 makepkg-mingw –sLf lines to MINGW_INSTALLS=mingw32 makepkg-mingw -sLf.

Get the tools

Note Josip uses a cx_Freeze with some changes he made to make the process a bit easier. I attempted to make similar changes, but we will have to build the utility.

Download https://gramps-project.org/wiki/images/a/a3/Mingw-w64-python-cx_Freeze.zip (stored here) to a convenient location with your browser.

I do my builds in the c:\msys64\home\user\builds directory, so we will make a new sub-directory for cx_Freeze there.

I use 7-zip to extract to the c:\msys64\home\user\builds\cx_Freeze directory.

Note Use MSYS2 MSYS shell.

 MINGW_INSTALLS=mingw32 makepkg-mingw -sLf
 pacman -U mingw-w64-i686-python3-cx_Freeze-5.0.2-1-any.pkg.tar.xz

The patches included in the zip should fix up cx_Freeze for a bug where it tries to load an sqlite3.dll file, and to add properly optimized base files for Python startup (In the original version the 'optimize' functionality doesn't work).

NSIS is a tool to build the actual installer. As of NSIS version 3.05 the standard nsis available from the MINGW archive works.

 pacman -S mingw-w64-i686-nsis

If you have NSIS version 3.04, The version in the mingw system has a bug that makes displaying of the font files during the AIO install messed up. I've created a patch, so we need to build our own. Download https://gramps-project.org/wiki/index.php/File:Mingw-w64-nsis.zip (stored here) to a convenient location with your browser.

Unzip and cd into the directory then:

Note Use MSYS2 MSYS shell.

 MINGW_INSTALLS=mingw32 makepkg-mingw -sLf
 pacman -U mingw-w64-i686-nsis-3.0.4-1-any.pkg.tar.xz

Get a reference AIO

We also need the most recent version of Gramps AIO available. If it is for your branch, it is likely to have the closest scripts with minimal editing. Of course a newer version for a newer branch might have some changes in the scripts that deal with more recent dependencies. You may want to reference both in updating scripts for the new version. Get it installed so you can see the files from that installation.

Install the old Gramps

When you start the installer, it will ask you to select language and spell check files. Select all of them. It will then ask where you want it installed. You can install it where you want, as it is only a temporary location which you may delete afterward. But make a note of the folder since you will need to know it later. Also, it needs to be a new (empty) folder, not an already existing one.

In this tutorial we'll say you tell the installer to put it into a new "C:\Program Files (x86)\GrampsAIO32-5.1.0" folder.

Then double-click on the AIO executable to start the 5.1.0 installation. (If your Windows asks whether you will allow that program to change things on your computer, say yes of course.)

You can put it anywhere you want but if you do, change the string "C:\Program Files (x86)\GrampsAIO32-5.1.0" in this tutorial to the actual name you used. But every time I use "C:\Program Files (x86)\GrampsAIO32-5.1.0" I mean the top of the tree where you installed the "old" AIO.

After the installer is finished, the last window will offer you the opportunity to "Run" the Gramps you just installed, so you should un-click that box before you hit "Finish", since there is no need to run the old one.

Similarly, you can delete the "GrampsAIO32 x.x.x" (or GrampsAIO64 one)" icon shortcut from your Desktop, since you won't be using it.

Get Gramps installed in our development environment

Note Use MSYS2 MINGW32 shell.

If you have not already done so as part of the prior steps, do the following:

 cd ~/grampsdev  # my copy of git repo checked out with the branch you are building.

 python3 setup.py build

To install preparatory to creating AIO do

 python3 setup.py install

Building the AIO

Note: Gramps contains some spell checking files that are NOT installed by the previous 'pacman' installs. I found it easiest to copy them from the existing Gramps installation (where I installed all the language and spelling files during the AIO install). The originals were in the installation /share/enchant/myspell directory. I copied the whole 'myspell' directory above to my c:\msys64\mingw32\share\enchant directory so they could be found by the next step scripts. I used the Windows file explorer as that is easiest for me, but if you are familiar with the Linux cp command, you can do it from the MINGW32 shell.

Before doing the AIO build, you may want to compare the existing src/setup.py and src/grampsaio32.nsi scripts with the latest ones available from other AIO installations. They change depending on the contents of the AIO and may point up other dependencies or changes. I use Notepad++ and its 'Compare' plugin for this, but feel free to use whatever tool you like.

I use dir ~/aio where my setup.py, grampsX.py, icons and nsis script is stored.

 cd ~
 cp -r "/c/Program Files (x86)/GrampsAIO32-5.1.0/src" aio

Modifying the setup.py script

The setup.py script should be located in ~/aio/setup.py. Use you favorite editor to make sure following is right:

• Python versions are correct; At the moment, Python3.6 is in use, however if your version is different, adjust the references to python3.6 and cpython-36m and python36 appropriately.
• The 32/64 bit differences are correct; gspawn-win64-helper may need to change to gspawn-win32-helper or vice versa.
• If you have added new dependencies, find the right place to put them into the script.

Then use the following to create ~/aio/mingw32 with all files needed for bundle.

Modifying the gramps.nsi script

Open the gramps.nsi file with your favorite editor.

Part of the contents should look something like the following:

!define APPNAME "GrampsAIO32"
!define APPVERSION "5.1.0"
!define APPBUILD "1"
!define APPNAMEANDVERSION "${APPNAME} ${APPVERSION}"
!define APP_PUBLISHER "The Gramps project"
!define APP_WEB_SITE "http://gramps-project.org"
BrandingText "Version ${APPVERSION}-${APPBUILD} -- a new maintenance release"

Make sure your editor works One user reported unicode problems with Notepad and then downloaded Notepad++.

Increase the version number

Change the APPVERSION line from "5.1.0" to "5.1.1" instead (or higher if making a newer version).

If needed update APPNAME and APPBUILD

Possibly APPNAME and APPBUILD will need changing too, depending on what you are trying to do. Generally the APPBUILD gets incremented if a second AIO release is made from the same version (because of a building issue.

If you are a power user and have downloaded something other than a tar.gz file, see Appendix A to change APPBUILD.

If necessary, change the BrandingText line to have the name of the new Gramps release instead of the old release name. Usually only necessary for major version changes.

Then save that gramps.nsi file.

Run the scripts

 python3 setup.py build_exe --no-compress

Then I pass nsis script from ~/aio/mingw32/src to nsis.

 makensis ~/aio/mingw32/src/grampsaio32.nsi

Note that after the compilation, the resulting files will be compressed.

Please be patient The the last part of the setup.py script and the NSIS package compression will probably take a while -- OVER FIVE MINUTES FOR ME --

Experimenting with NSIS compression See Appendix B for power-user details, at this stage.

The result is in C:\msys64\home\user\aio\mingw32\src\GrampsAIO-5.1.1-1_win32.exe.

If you do this more than once, make sure to delete the result file, or it will get encoded into the next AIO.

Debugging

I had several issues in this process. First of all, the mingw repositories are "rolling release", so every time you do a pacman –Syuu you will have different packages and potentially new issues.

Take care to make sure that you don't overwrite any of the 'static' Gramps dependencies (gtk 3.18.9, bsddb 6.1.0, db 6.0.30 at the time of this writing) with newer versions, and if you do, use pacman -U file.tar.xz to revert them. Get this right FIRST, because if you build anything based on one of the static dependencies, it may end up not working on a newer version. If you do build something out of order, you may end up with mysterious import failures with messages that don't tell you what is really wrong. And it's really hard to debug the installed AIO, as it doesn't look like a typical Python process to the usual debuggers. Experience speaking.

You may have to build one or more dependencies "the hard way"; that is download the PKGBUILD and any patches from a suitable source https://github.com/Alexpux/MINGW-packages/trunk/mingw-w64-osmgpsmap-git for example, and fix it up for newer versions and build it. See https://github.com/msys2/msys2/wiki/Creating-packages for help here. Or even download source code and compile yourself.

I used the old DOS tree /F /A > aionew.txt command to make filename dumps from both the new AIO and a recent older one that was freshly installed in an empty directory. This ensures that there are no '.pyc' files to confuse the issue. Then I did a difference compare (using Notepad++) of the two to see what files are different. Any missing files in your new AIO are suspect (unless you did this deliberately).

Appendix A

Indicating newer APPBUILD's

If you are a power user, you might have downloaded or used git to checkout the latest code in the grampsxx branch:

 https://codeload.github.com/gramps-project/gramps/zip/maintenance/gramps51

but in any case, if you have downloaded something other than a gramps-release tar.gz (or .zip) file, say this in your top directory ~/grampsdev (assuming you have installed Git for Windows):

 git rev-list --count HEAD

and then

 git rev-parse --short HEAD

and then put an "r" in front of the first string, followed by a hyphen ("-") and the second string, and then use the resulting combined string as your APPBUILD in the gramps.nsi file, for clarity.

So for example your APPBUILD line might look like this:

 !define APPBUILD "r23381-25018d6"

Then continue with the NSIS package building

Developers uploading to Sourceforge

If you are a developer and plan to upload it to SourceForge and it is an AIO of (a forthcoming) "master" put it into:

 http://sourceforge.net/projects/gramps/files/Unstable

but if it is an AIO for a maintenance branch, make a Snapshots folder in that branch and put it there, e.g.:

 http://sourceforge.net/projects/gramps/files/Stable/5.1.1/Snapshots

so then the APPBUILD numbers will uniquely define that AIO.

But note that the NSIS-generated filename will say (for instance) "5.1.2" even though it is really a "5.1.1" with bugs fixed, which is why it belongs in 5.1.1/Snapshots.

Also see What to do for a release

Appendix B

Experimenting with NSIS compression

If you are a power user and are experimenting with slight changes to the gramps.nsi script, you can drastically shorten the time needed to have NSIS generate an AIO package by commenting out these two lines (with a semicolon, a ";"):

 SetCompressor /SOLID /FINAL LZMA
 SetCompressorDictSize 128

But if you are building an AIO which will be released to the public (put onto the official gramps files repo), you need to be very sure to uncomment both those lines first.

See also

• All In One Gramps Software Bundle for Windows for the Gramps 3.4.x and earlier series
• Gramps for Windows with MSYS2
• GrampsAIO cx freeze-based