There’s been questions here and there asking why Wikipedia/the Wiktionary haven’t been updated in a while. It’s been mentioned in the newsletter already but this post is more of an evergreen place and it does not hurt to spread the word.
As the title says, we are currently in the process of rewriting the entirety of mediawiki offliner, which is the scraper behind every single offline iteration of any wiki available on Kiwix (starting with Wikipedia). The work is hard, and it is probably even harder because we’re a small org with limited resources (there is no employee toiling on it every day, because simply put we do not have the money for a full time maintainer).
Why now?
Last Fall the Wikimedia Foundation made an number of changes to its API that impacted mwoffliner’s ability to manipulate their dumps. If you are not into coding, now is a good time to be introduced to the concept of regression, meaning that things that used to work do not anymore, and nobody saw it coming (as opposite to “this change will break that thing here, so let’s fix it ahead of time”, which we did with our good friends at the WMF). There were a number of regressions that appeared, and we decided it was time not for a fix, but an actual overhaul: mwoffliner has been in version 1.xx since its inception ten or so years ago, and the tech has moved enough that we finally decided to make the jump to version 2.0.
What now?
There’ll be a small jump to version 1.14 first, and then onwards to 2.0. Work started about a year and a half ago. It has cost us money we did not always have, and having ten years’ worth of code to revamp/rethink is no small endeavour. We have already spent 2 (3?) contractors on the task already. The larger wikis (like the English Wikipedia, maxi version) have stopped being updated at the beginning of the year, though some nopic or mini versions may still be running (images are trouble).
We’ve struck gold however just before Summer, when a volunteer at the Spring hackathon turned out to have the right ideas and, more importantly, the right skillset to put them into place. He has, however, a real job (and so do those doing the code review), so things have gone their own (slow) pace for the past two months. There were also other projects that needed finishing and that were more time-bound (mentoring the Google Summer of Code students, managing the handover for someone going on a long paternity leave, etc.).
What next?
These other projects are now past us and it looks like things should start to pick up pace again \o/
As you know, Kiwix is free and open-source, and open is the key word here: if you really want to keep an eye on where we’re at (and cheers us on), there are five core issues that remain on the project list and that you can follow directly on github: #1576, 1974, 1979, 2000, 2007 (you can actually bookmark them to get notified of changes; you can also “star” the project, though I am not entirely sure of the added value besides internet points).
After that there’ll be some more bug fixing, testing, and running the actual updated scraper. Though we can’t really commit on a release date, I’m told that the End is Nigh we are seeing the end of the tunnel.
Kiwix is a non-profit, sells no ads and tracks no data. If you find it useful (and if you have read this wall of text, why wouldn’t you), consider making a donation to help us keep working on such great projects.