Daniel Lange: Upgrading Limesurvey with (near) zero downtime

Limesurvey is an online survey tool. It is very powerful and commonly used in academic environments because it is Free Software (GPLv2+), allows for local installations protecting the data of participants and allowing to comply with data protection regulations. This also means there are typically no load-balanced multi-server szenarios with HA databases. But simple VMs where Limesurvey runs and needs upgrading in place.

There’s an LTS branch (currently 3.x) and a stable branch (currently 4.x). There’s also a 2.06 LTS branch that is restricted to paying customers. The main developers behind Limesurvey offer many services from template design to custom development to support to hosting (“Cloud”, “Limesurvey Pro”). Unfortunately they also charge for easy updates called “ComfortUpdate” (currently 39€ for three months) and the manual process is made a bit cumbersome to make the “ComfortUpdate” offer more attractive.

Due to Limesurvey being an old code base and UI elements not being clearly separated, most serious use cases will end up patching files and symlinking logos around template directories. That conflicts a bit with the opaque “ComfortUpdate” process where you push a button and then magic happens. Or you have downtime and a recovery case while surveys are running.

If you do not intend to use the “ComfortUpdate” offering, you can prevent Limesurvey from connecting to http://comfortupdate.limesurvey.org daily by adding the updatable stanza as in line 14 to limesurvey/application/config/config.php:

The comma on line 13 is placed like that in the current default limesurvey config.php, don’t let yourself get confused.
Every item in a php array must end with a comma. It can be on the next line.

The basic principle of low risk, near-zero downtime, in-place upgrades is:

1. Create a diff between the current release and the target release
2. Inspect the diff
3. Make backups of the application webroot
4. Patch a copy of the application in-place
5. (optional) stop the web server
6. Make a backup of the production database
7. Move the patched application to the production webroot
8. (if 5) Start the webserver
9. Upgrade the database (if needed)
10. Check the application

So, in detail:

1. Create a diff between the current release and the target release

For Limesurvey it is essential that you keep a version of the currently running release before you have applied your templates and other customizations.
If you missed that step or are doing the upgrade like this the first time, the Limesurvey github releases page is a useful source.

In this example we are upgrading from release 4.2.6 to 4.3.0. So we skipped 4.2.7.

I assume you’re working in a /root/install directory for this step. You should definitely work outside the webroot.

You should have a 4.2.6 directory containing the unpacked Limesurvey 4.2.6 zip or tarball.

Download the 4.3.0 release, mkdir 4.3.0 and (for example) unzip -d 4.3.0/ limesurvey4.3.0+200616.zip.

Create the diff with
# diff -N -u -r 4.2.6/ 4.3.0/ > mydiff

-N allows to create new files as a diff from /dev/zero and to remove files similarly


-u creates a “unified diff”, so a diff with some context that will allow to apply patches even to files that you changed locally


-r makes the diff recurse through subdirectories

2. Inspect the diff

As a first step look at the size of the diff:

$ wc -l mydiff

49896 mydiff

That looks reasonable, it’s a web app .

If you get 8.5m lines, you have managed to get one directory wrong and the whole app is removed and re-created.

Now read through the diff and inspect for changes that may collide with your local modifications or that look fishy.

A list of the files that got created or removed may be useful to inspect at minimum:

$ diff <(cd 4.2.6; find . -type f) <(cd 4.3.0; find . -type f)

2185,2188c2185,2188

< ./limesurvey/third_party/jquery/jquery-migrate-3.1.0.js

< ./limesurvey/third_party/jquery/jquery-3.4.1.min.js

< ./limesurvey/third_party/jquery/jquery-3.4.1.js

< ./limesurvey/third_party/jquery/jquery-migrate-3.1.0.min.js

—

> ./limesurvey/third_party/jquery/jquery-3.5.1.min.js

> ./limesurvey/third_party/jquery/jquery-3.5.1.js

> ./limesurvey/third_party/jquery/jquery-migrate-3.3.0.min.js

> ./limesurvey/third_party/jquery/jquery-migrate-3.3.0.js

10014a10015,10016

> ./limesurvey/locale/ti/index.html

> ./limesurvey/locale/ti/ti.mo

10058a10061

> ./limesurvey/tests/data/surveys/survey_archive_265351_listParticipants.lsa

10079a10083

> ./limesurvey/tests/data/surveys/survey_archive_821351.lsa

10115a10120

> ./limesurvey/tests/unit/helpers/QuexmlPDFTest.php

10120a10126

> ./limesurvey/tests/unit/helpers/RemoteControlListParticipantsTest.php

12241d12246

< ./limesurvey/application/controllers/admin/questionedit.php

12277a12283

> ./limesurvey/application/controllers/QuestionEditorController.php

12835,12836d12840

< ./limesurvey/application/views/admin/survey/Question2/view.php

< ./limesurvey/application/views/admin/survey/Question2/_jsVariables.php

13414a13419,13420

> ./limesurvey/application/views/questionEditor/view.php

> ./limesurvey/application/views/questionEditor/_jsVariables.php

13438a13445

> ./limesurvey/application/views/layouts/title_bar.php

13442a13450,13451

> ./limesurvey/application/views/layouts/sidemenu.php

> ./limesurvey/application/views/layouts/layout_questioneditor.php

3. Make backups of the application webroot

cd /var/www

cp -ra limesurvey limesurvey_backup_${date “+%y%m%d”} # create backup

cp -ra limesurvey limesurvey_new # create tree to patch

4. Patch a copy of the application in-place

cd limesurvey_new

patch –verbose -N -p 2 < /root/install/mydiff

The --verbose is important so you see deleted files. -N forces a forward-patch as everything else would not make sense.
-p 2 strips two directories off the beginning of the paths in the diff. That makes sense as 4.3.0/limesurvey is not needed when you have limesurvey_new as your working directory.

The output will look like this:

If you see any hunks rejected (.rej files created), find the reason and fix it before continuing.

5. (optional) stop the web server

If you run a stupid webserver, a complex setup, external opcode caches or just have high load (survey writes likely), you may want to stop the httpd now.

For “normal loads” (a few hundred answers to surveys per day) and when using Apache you can typically just move the webroot.

In any case even with stopping the httpd, the downtime should be very short.

6. Make a backup of the production database

Make a backup of the production database. In our case this is as simple as calling backup-mysql.

If you don’t have a specific way to backup your database

mysqldump --quick --extended-insert --skip-comments limesurvey | gzip -9 > /root/install/limesurvey_database_${date "+%y%m%d"}.sql.gz

should be a good start.

7. Move the patched application to the production webroot

mv limesurvey limesurvey_old && mv limesurvey_new limesurvey

This will overwrite the old webroot and is two fast atomic operation on sane filesystems (e.g. ext4), so this is rather safe™ to do.

You can as well patch the production webroot in place. Esp. if you have stopped the httpd. You should have made a backup in step 3. You did, didn’t you?

Note: If you want a fully atomic replacement of the application webroot, you have to use symlinks for the old and new tree and use mv -T, see Richard Crowley’s Things UNIX can do atomically for a discussion.

8. (if 5) Start the webserver

If you stopped the webserver, start it again.

9. Upgrade the database (if needed)

Depending on whether the upgrade needed database schema upgrades, you want to inspect the Admin UI at:

https://limesurvey.example.org/admin

The 4.2.6 -> 4.3.0 upgrade does not need any database changes.

10. Check the application

As a good last step – as always – take a critical look at the application whether everything runs as it should.
Are the images loading, can you do a test survey?

If so, all fine and until the next upgrade.