Wikipedia in Your Pocket

aka. The Hitchhiker's Guide to the Galaxy, v1.0 :)

This Instructable will set out how to construct what I believe to be a unique implementation of Wikipedia in an offline, portable device. It involves installing a stripped-down distribution of Linux on a Psion 5mx handheld, and installing a static HTML version of Wikipedia for use with one of two browsers. Most importantly, you do not have to be a Linux wizard to achieve this. I will assume a basic familiarity with computers, but you do not need experience with the intricacies of filing systems, compiling source code and the stuff that traditionally puts people off using Linux. Following the steps in this Instructable should let you create your own HHGTTG- I will explain what I am doing but also try to give as explicit instructions as possible.

(For this reason, seasoned linux hackers can take the "skip this step" instructions provided at the top of the steps involving partitioning the CF card and unpacking archives to it)

Also, I apologise deeply for how dry this Instructable is. I have tried hard to make the bulk of it readable without skimping on detail or being overly prolix, but explaining how to partition a removable volume and unpack .tgz archives to it is difficult to make amusing. Sorry.

Lastly, but by no means least, I could not have done this without the efforts of:

• the people who ported Linux to the ARM,
• the OpenPsion (née PsiLinux) community for porting Linux to the Psion,
• Adrian Wells for Kludged Linux, and of course
• all the good folks at Wikipedia.

My hack pales into insignificance compared to any of these achievements. (This lengthy grovel is mostly lest anyone accuse me of taking credit for the work of others. I emailed Adrian on the 22-04-2008 to ask if he objected to me reproducing his works here, but had no response)

Step 1: Required Materials

The major sticking point for this Instructable is that you require a working Psion 5mx. These are comparatively rare, but by no means unobtainable- a quick ebay search revealed 16 on the first page of a search for "psion 5mx". They are not cheap, usually running to around £70, but you may well find one for less, or even free if a friend has one they no longer use.
The other parts can be easily obtained more cheaply- I spent about £15 for all the parts except for the Psion, which was a gift.

You will need: (with approximate cost)

- A working Psion 5mx PDA. Around £70 (ebay), or possibly free if you search enough attics.
- A Compactflash card of at least 1GB- I recommend a 4GB card. £11 (ebay)
- A Compactflash card adapter for your PC, either USB or PCMCIA. £4 (ebay)
- A PC running Linux. If you don't have it installed, I would suggest burning a live CD or making a bootable USB drive for a distribution such as Puppy.
- The archives with which to install Kludged Linux on your Psion. These may be found at Adrian's site here, at the bottom of the page under "download". If his site should disappear, let me know and I will provide my copies for download
- Static HTML dump of Wikipedia.
Text-only dump from 2003 (large file download)
CD distribution of 4,500 good articles with thumbnail images (warning, 750MB file download)
There is a static dump of the entire English language Wikipedia as of June 2008 here: http://static.wikipedia.org/downloads/2008-06/en/
- A flask of weak lemon drink

Step 2: Find the CF Card Device

Before you start, it goes without saying that you should back up any important information that may be on the Psion. It is also possible (although unlikely) to format your computer's hard disk during this step, but this should not be a worry if you take sufficient care during the formatting stage. Also, if you are using Puppy it shouldn't even mount your hard disk unless you explicitly tell it to.

For Linux wizards- to skip this step, determine the device ID (eg /dev/hde or /dev/sdb) of your CF card, ready to partition in the next step

Open a console, and determine what device the CF card is mounted as. You can do this by typing:
cd /dev
ls -l > ~/devices1

Then plug the CF card adapter into your PC, and type:
ls -l > ~/devices2
cd ~
diff devices1 devices2

You will see a line or two beginning with > with no matching line beginning with <. The cryptic few letters are what we are looking for. See image 1
Type
cd /dev
ls xxx
where xxx are the letters you found earlier- these are likely to be hde, sda, sdb or similar. Now plug the CF card into the reader and type
ls xxx*
You should see, in addition to the one you saw earlier (eg "sdb") another entry with a number after it, eg "sdb1". This represents the partition on the CF card, whereas the entry with no numbers (sdb) represents the card itself. See image 2

Step 3: Partition and Format the CF Card

To skip this step: create primary partition 1 as an 8MB, bootable MSDOS (type 4) partition, and primary partition 2 as non-bootable Linux (type 83) taking up the rest of the card

For this step you will require admin privileges on the Linux machine. The easiest way to do this is type su and the root password, but if you're overly concerned or don't have root access you can type sudo for the important commands and give your user password.

Type fdisk /dev/XXX where XXX is the device identifier we found earlier. From now on I will refer to it as XXX.

Type p to print the partition table- if an entry such as this appears, type d to delete it. Type p to check there are no remaining partitions- if there are, type d again to remove any remaining ones. See image 1

Now type n for a new partition, p for primary, 1 for partition 1, 1 to start it at the beginning of the disk, 8M to make it 8MB in size
a then 1 to make it bootable
t then 4 to make it a type 4 (DOS) partition. Type p again to check your current partition table. See image 2

Now type n for a new partition, p for primary, 2 for partition 2, 9 (or the lowest number available) to start it after the end of the DOS partition, 1023 (or the highest number available) to make it fill the remaining space
t then 83 to make it type 83 (Linux)
p again to make sure your partition table looks like picture ####.

w to write your changes to the card, then q to quit.

Now if you type ls /dev/XXX* again you should see XXX for the CF card, XXX1 for the DOS partition and XXX2 for the Linux partition. If this happens- congratulations, you've done the most difficult part of this Instructable. Drink deeply from your flask of weak lemon drink to congratulate yourself on your victory of the machine.

Step 4: Set Up the Filesystem

To skip this step, create a DOS filesystem on partition 1, and an EXT2 on partition 2. Put a file on the DOS partition, then plug the card into your Psion and start it up to make sure it is happy with the partitioned CF card

Type mkfs.msdos /dev/XXX1 and then mkfs.ext2 /dev/XXX2 . Now you are ready to make sure your Psion is happy with the CF card.

Type cd /mnt
mkdir psiondos
mount -t vfat /dev/XXX1
/mnt/psiondos
cd /mnt/psiondos
touch hello.txt
to mount the card on your PC, and create a file on the DOS partition, then
cd /
umount /dev/XXX1
to unmount the card.

Now plug the card into your Psion and turn it on. Use the button in the bottom left of the screen to navigate to the D drive- you should see your file hello.txt in the file manager. If you do, then the Psion is happy with your card and you can proceed to install Linux on it. If not, either there was a problem with creating the filesystem in the previous steps or your Psion does not recognise the card- try repeating the previous steps, getting a friendly Linux wizard to set up the file system for you, or (as a last resort) try with a different CF card to see if you can find one that works.

Step 5: Install Kludged Linux

To skip this step, untar the Kludged Linux archive ksf_.tgz to the Linux partition, and the contents of the files directory in the files.zip archive to the DOS partition

Plug the CF card back into your PC, and repeat the
mount -t vfat /dev/XXX1 /mount/psiondos step.

Now type
cd /mnt
mkdir psionlinux
mount -t ext2 /dev/XXX2 /mnt/psionlinux
to mount the second partition.

Now you need to unzip the contents of the files.zip archive, and copy the contents of the files directory into /mnt/psiondos. One way to do this is
cd ~
unzip /path/files.zip (where /path/files.zip is the path to where the zip archive is saved)
cd files
cp -r * /mnt/psiondos

Now you need to unpack the Kludged Linux distribution to the Linux partition. Type
cd /mnt/psionlinux
tar -xzvf /path/ksf_.tgz (where /path/ksf_.tgz is the path to the Kludged Linux archive, eg , eg ~/downloads/ksf100807.tgz)

This will take a few minutes, so drink your flask of weak lemon drink while pondering life's imponderables.

Step 6: Install Your Wikipedia Dump

To skip this step, unpack your dump of Wikipedia to somewhere sensible on the Linux partition, for instance /usr/Wiki

How you do this step will depend on which Wikipedia dump you are using. I will show the process for the CD distribution.

You can just unpack the archive straight to the CF card, but in my experience there may be some teething troubles with the installation of Linux which require clearing and rewriting to the CF card, so in the long run it is more sensible to unpack to your hard disk and copy the completed dump to the CF card.

Create a directory for your dump to be unpacked into, for instance by typing
cd ~
mkdir wikidump

Unpack the archive into this directory:
cd wikidump
unzip /path/school-wikipedia-small.zip

This will take a while, depending on the size of your dump and speed of your computer. I suggest you take this opportunity to refill your flask of weak lemon drink, adding flavour enhancers such as ginger, lime juice or vodka to suit your taste. Once the archive has finished unpacking, find the directories you require- for the CD distribution these are wp andimages. Copy these to the Linux partition in the /usr directory:
cp images wp /mnt/psionlinux/usr/

This will take a long time- upwards of 10 minutes for the CD dump which is around a gigabyte in size. Take your flask of weak lemon drink to the brow of the nearest hill and gaze towards the horizon, imagining the untold numbers of things you will do with your new Wikipedia-in-your-pocket, such as travelling the country hustling pub quizzes. Return from your dreamlike reverie and return to the computer to see your dump finish copying to the CF card. From the CD version you will also need to copy the index.htm file from the root of the archive:
cp index.htm /mnt/psionlinux/usr/

You are now ready to put the card in the Psion and get going!

(Note: I used both the Terodump and the CD version, because the Terodump has a broad coverage of short text articles, whereas the CD version has more up-to-date, detailed articles with images about important topics, so the two together have both breadth and depth. Space constraints on your CF card may limit your choice of Wikipedia dump, in which case you will need to choose the version that best reflects your desired usage.

If you are installing two different versions, I suggest putting each in its own directory, for instance /usr/terowiki and /usr/cdwiki). What you copy across will depend on the wikipedia dump you are using- from a Terodump you basically want the "wikipedia" directory and everything in it, and maybe the "index.html".

Step 7: Put It Together

You are now ready to start using the Psion. Unmount the CF card:
umount /dev/XXX1
umount /dev/XXX2

Unplug the CF card and plug it into the slot on your Psion. Turn on the Psion, navigate to the D drive, and double click on arlo.exe to run it.

The Psion should present you with a boot menu- you can press 0 to boot up or wait 5 seconds for it to continue. You will then see a penguin logo and a lot of text scrolling past as Linux starts up. The boot sequence may complain of error in the filesystem- unless these are serious enough for the boot sequence to stop, they can usually be ignored. You will eventually see a boot prompt saying
Debian GNU/Linux 3.1 tty1
(Root password is "root")

Psion login:

Type root as username and password, and you will reach a command prompt (root@Psion: ~#). See image 1.

Congratulations- you now have a very small and quite underpowered Linux palmtop. Drink a toast to your success with the remainder of your weak lemon drink. However, the command prompt is not a very useful place to be if you want to browse Wikipedia. Type xinit to start the X window system. You should see the screen turn blank for a moment, then a stippled pattern with a black X in the middle of it. If the screen remains blank for more than about 10 seconds, go to the bottom of this step to find the "No X startup" fix.

After the stipple background shows for a few seconds you should see a taskbar at the bottom of the screen- this consists of a CPU usage meter (useful for telling whether the Psion is thinking hard or not), a Menu button that holds the applications, and a clock (that will probably be wrong- ignore this).

See image 2

The parts below are for information purposes- if you are happy at this point in the install you can continue to the next step.

To turn it off:
To quit X, press Ctrl-Menu-Backspace to return to the command prompt. To turn off the Psion (for instance to change batteries or add additional content to the CF card), type shutdown -r now at the console, and press Menu-Esc after the Psion beeps, while the screen is blank or displaying the boot menu but before the penguin appears. You can now safely remove the CF card and batteries- when you want to use the Psion again, replace these and turn it on, and it will boot up Linux. Note that you do not have to shutdown the Psion to turn it off- just press Menu-Esc to make it sleep, and press Esc or tap the screen to wake it back up. It is only necessary to shut it down to replace batteries or the CF card.

To go back to "being a Psion":
To return to the Psion's EPOC operating system (but why would you want to when you have Linux?), turn the Psion off as above. Remove the CF card, and turn the Psion on. It will give three falling beeps, and return to the PSION Series 5 startup screen. To return to Linux, you will need to navigate back to arlo.exe on the D drive and run it.

No X startup?
I have found that occasionally, especially if the Psion complains of errors in the file system, it will refuse to start the X window system. If the xinitmmand just shows a blank screen, press Menu-1 to get back to the console. If it says "Touchscreen not found" or something similar, shut down the Psion as described above and restart it- this fixed the probem every time it occurred for me. Also, since running e2fsk on the linux file system while teh CF card was mounted on my PC has stopped the Psion complaining of file system errors, and the "Touchscreen not found" bug has not reoccured. YMM, as always, V

Step 8: Set Up Dillo

Tap the menu button, and select Net -> Dillo from the menu that pops up. From this point onwards you will need patience, many GUI processes take quite a while running on the slow Psion processor.

Dillo is just a lightweight web browser, so can be used like any otther (type an address in the address bar, click on links to activate etc.). To be useful on the Psion screen, however, it requires a few tweaks.the V menu and select "Options...". This will take around 30 seconds to load, with little indication that it is happening other than the CPU graph, but you will eventually reach an options screen. Tap the "Font" tab and set the font factor to around 1.4 depending on how large you want text to be (2 gives very large, clear text, 1.2 gives smaller text but allows more to fit on screen).

If you can handle another lengthy-ish wait, tap the Color tab and the Link color block. After another pause you will see a color selection screen- I suggest setting the value slider to a low value to make links more visible on the LCD.

Finally, under the Network tab you will find the start page and home page addresses- I suggest setting both of these to the path of the index page of your Wikipedia dump, as it will appear on the psion's file system, prefixed with file:// because it is not a web address (For example file://usr/index.htm)

Once you are happy with the settings, click the Save button, quit Dillo and restart it from the menu. It should start up to your Wikipedia dump index page.

Step 9: Using Dillo

Dillo is easy to use to browse Wikipedia with- the scrollbar on the right-hand side, home button at top left and links within pages are all you are likely to need. The touch screen defaults to delivering a left-click. Pressing Ctrl-1, Ctrl-2 or Ctrl-3 will set it to deliver left, middle or right clicks accordingly, so pressing Ctrl-2 before clicking on a link will open the link ina new tab. (I don't recommend this, for the record, because the Psion is slow enough with one tab open!). Remember that these are persistent modifiers, so after changing it every click will be a middle- or right-click until you set it back to left click with Ctrl-1.

/ or Ctrl-F can be used to search for text within a page, although if Dillo is running in full-screen mode the search term text box will not be shown. More on how to use Dillo can be found at their homepage here at www.dillo.org.

Now you can go to the index page, search for, for instance, lemons, to find out all about your favourite hobbyist's beverage.

Step 10: Optional: Text Browser

Dillo is very easy to use and gives the option to view pictures, but suffers from slowness and the fact that the text is graphically rendered so not as sharp as the text in character mode. If you would prefer a text-mode browser for faster browsing, sharper text and general hairy-chested text-mode Linux wizard points, Kludged Linux includes the dpgk utility to install .deb packages. Links is a text-mode browser with a very low memory footprint and few dependencies, making it ideal to install on the Psion. The deb package can be found on the debian old stable package repository. To install links:

Download the ARM architecture links package to your PC from one of the mirror links on this page
Shutdown the Psion (shutdown -r now at the console, then press Menu-Esc after the beep before it boots up again) and remove the CF card to plug into your PC
Mount the CF card (mount -t ext2 /dev/XXX2 /mnt/psionlinux)
Copy the file to the CF card (cp /path/links_blah.deb /mnt/psionlinux/usr/)
Unmount the CF card (umount /dev/XXX1), plug it back into the Psion, turn it on and wait for it to reboot.
Now, after logging in, type
'cd /usr
dpkg -i links_blah.deb (tab completion helps out here, you can just type "dpkg -i links" and press tab to let the prompt complete the command for you
links index.html (or whatever the path to your index page is) will start links at the chosen page. Again, sorry there are no screenshots but IIRC this process doesn't give you much feedback if everything works OK, so there's not much to show.

Links takes a little getting used to, especially as it scrolls page by page rather than line by line. The important commands in Links are:
space- down a page
b- up a page
down arrow- next link
up arrow- previous link
right arrow- follow highlighted link
left arrow- back
esc- open menu (while in menu, arrow keys navigate as expected)

Again, more about how to use Links at their website.

Step 11: You're Done!

Congratulations- you now have an object of beauty, utility and massive geek bragging rights. Show it off to appreciative audiences of hackers and Douglas Adams fans. Take it to a pub quiz and be ejected. Be the guy who has the answer to obscure and irrelevant questions.

And most importantly..

Don't panic!

Step 12: In Which the Author Solicits Contributions

Firstly, thanks to everyone for your positive comments! It means a lot to hear people "sass" my work :)

Secondly, it has come to my attention that a fair few people have different hardware they would like to try this on, but can't get their hands on a 5mx.

Using a Psion makes it difficult.

The original Psion operating system was no good because it didn't have the ability to display HTML files, which ultimately are what Wikipedia is made of, so installing Linux was necessary to get this feature. There are plenty of modern PDAs and smartphones that already have a browser, some even running Linux out of the box, that would make great bases for a similar project.

If your PDA or smartphone has a browser, all you have to do is get a large memory card that your device can read from, and unpack a static wikipedia dump to it.

If you have a Nokia 9300, you don't need to install Linux- it already has a browser with the Symbian OS it's running, so all you should have to do is plug in a memory card with the wiki dump on it.

If you have your own device and are thinking of doing something similar, I would love to see photos or screenshots of it running- if you're happy for me to do so I will add them to this Instructable and credit you