July 27, 2020

Offline Wikipedia on the Kobo

I often use my Kobo Clara HD where there is no WiFi available and the included offline dictionaries aren't always sufficient. So it would be nice to be able to look up words or names of people mentioned in the book on Wikipedia. With an internet connection the default Kobo firmware allows you to do this by long-pressing and then selecting the Wikipedia icon. So it should be possible to setup Kiwix with an offline copy of Wikipedia and use that instead. As it turns out it is possible with a few hacks. After a little research I found that a3nm did something similar in a blog post from 2013, which is a bit outdated now though.

This article assumes that you have access to the Kobo's internal files, either through a root shell using SSH or Telnet or by taking out the SD Card and that you are using a Unix operating system. Before doing anything make sure you have a backup.

Setup

You will need to download a Wikipedia ZIM-Archive of your choice from the corresponding page. I used the English "all nopic" version, which is 38GB as of writing this and will therefore require an SD-Card upgrade.

The naming scheme is explained in Kiwix's ZIM Readme:

"mini" means that only a subset of the text is available, probably the first section

"nopic" means that most pictures have been removed

"maxi" means everything except large media files like video/audio

no indication means we have done our best to scrape everything

You will also need to grab the latest version of kiwix-tools from the kiwix download page, which includes the kiwix-serve program that we will use to run our own local Wikipedia clone. Copy the contents of the sd card to the new one using dd and create a new partition with the remaining space using fdisk, for example. This is where we will put the ZIM archive.

dd if=/dev/old of=/dev/mmcblk0 bs=4M status=progress
fdisk /dev/mmcblk0
...
mkfs.ext4 /dev/mmcblk0p4

Extract kiwix-tools to /big and for convenience add

::sysinit:/bin/sh /opt/afterinit.sh

to /etc/inittab and put this in /opt/afterinit.sh:

ip link set lo up
sleep 3
mount /dev/mmcblk0p4 /big
sleep 2
/big/kiwix-tools_linux-armhf-3.1.2-1/kiwix-serve --port=4444 /big/wikipedia_en_all_nopic_2020-05.zim &

This will first start the loopback interface, which was disabled by default on my device. Then after waiting a bit it will mount the previously created partition to /big, which you will have to create first with mkdir /big. After sleeping for a bit again it starts the Kiwix server listening on port 4444. You'll have to adjust the ZIM path if it doesn't match.

Patching the firmware

For whatever reason, Kobo doesn't allow the browser to open if there is no WiFi available. So we will need to patch this by reverse engineering libnickel.so. I will go through the process of reverse engineering it in this article and then provide a patch at the end that might or might not work depending on your firmware version.

First get a local copy of libnickel.so:

scp root@kobo.lan:/usr/local/Kobo/libnickel.so.1.0.0 libnickel.so

Note: In the screenshots you can see Ghidra, which I mainly used for reverse engineering because of its great decompiler. However it seems that Ghidra's ARM assembler is a bit buggy, so I used radare2 for patching. The base address for all addresses mentioned here is 0x0 as opposed to Ghidra's default 0x100000.

At 0x00cef7a6 in WirelessWorkflowManager::connectWireless we need to patch a jump that checks whether the device is connected to WiFi and make it an unconditional b instead:

WirelessWorkflowManager::connectWireless disassembly

We additionally patch the call to showAirplaneModeDialog at 0x00cef818:

Call to showAirplaneDialog

and make it instead jump to internetIsAccessible at 0x04ee1cc:

internetIsAccessible

With these patches the Firmware will allow the browser to open in Airplane mode. All there is to do now is to make it use our local Kiwix server. For that we simply change two hardcoded strings:

One at 0x0ee3978:

First hardcoded string

and one at 0x0eef9e8:

Second hardcoded string

I have changed them to http://%1.wikipedia/wiki/special:Search and http://%1.m.wikipedia/wiki/special:Search, respectively. It is important to replace the https part with http, as we don't want to worry about TLS.

Or alternatively if you don't want to do it manually you can just try applying this patch using bspatch:

bspatch libnickel.so libnickel.so.patched wiki.patch

Upload the new version of libnickel.so to the Kobo:

scp libnickel.so root@kobo.lan:/usr/local/Kobo/libnickel.so.1.0.0

Then simply add this to your /etc/hosts file:

127.0.0.1 en.wikipedia en.m.wikipedia

Wikipedia search API redirector

The Kobo makes use of Wikipedia's wiki/Special:Search API to find an article for the selected text. Kiwix doesn't implement this API. For this reason I wrote a little server in Go that redirects search queries to corresponding Kiwix pages.

package main
import (
	"net/http"
	"strings"
	"flag"
)

func main() {
	listen := flag.String("listen", ":80", "Address to bind to")
	wikiurl := flag.String("wikiurl", "http://localhost:4444/wikipedia_en_all_nopic_2020-05/A/", "Wikipedia base url")
	removeTrailing := flag.Bool("removeTrailing", false, "Whether to remove trailing special characters")
	flag.Parse()
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		query := r.URL.Query()
		search := query.Get("search")
		if (*removeTrailing) {
			search = strings.TrimRight(search, ",.;[]\"'-")
		}
		http.Redirect(w, r, *wikiurl + strings.ReplaceAll(strings.Title(search), " ", "_"), 301)
	});
	http.ListenAndServe(*listen, nil)
}

Compile and move it to /big:

GOOS=linux GOARCH=arm GOARM=5 go build -o redirector
mv redirector /big

Then append this to the /opt/afterinit.sh script and set the wikiurl flag if it doesn't match your kiwix server.

/big/redirector -removeTrailing &

Reboot and that's it! The Kobo will now look up localhost/wiki/special:Search?search=<query>, where the Go redirector will turn <query> into a proper Kiwix Wikipedia article URI and redirect the browser there!

Kobo screenshot of Wikipedia page

This is what such an article looks like. The colors get greyscaled on the monochrome display, of course.

Finally, it would be nice to know how much more battery running the kiwix server uses. But I don't know how to monitor this and from normal usage I can't really tell a difference. The only difference is that when booting up (but not when waking from sleep) the Kobo is a little slow for a few seconds while kiwix-serve starts.