Tidying up my repository

Ive tidied up my repo a bit, if you are using ubuntu hardy, I have packages for amd64. To use them, add

deb http://apt.workbuffer.org/ hardy all
To your /etc/apt/sources.list

For Debian unstable, I only have i386 packages
deb http://apt.workbuffer.org/ unstable all

Then
sudo apt-get install cpan-libkoha-perl
If you want to try my expiremental, Koha package, or if you just want to install the bundle of Koha dependent modules, to make the rest of the install easier, then.

sudo apt-get install cpan-libkoha-support-bundle-perl

Making installing Koha dependencies easier

I did some work with CPANPLUS specifically using cpan2dist and CPANPLUS::Dist::Deb to create a distribution using Bundle::KohaSupport as the base.

So now we can do this, add a line to our sources.list
deb http://apt.workbuffer.org/ unstable main

run apt-get update
Then

chris@ubuntu:~/git/koha.git$ sudo apt-get install cpan-libbundle-kohasupport-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
cpan-libalgorithm-checkdigits-perl cpan-libansicolor-perl cpan-libbiblio-endnotestyle-perl cpan-libcgi-perl
cpan-libclass-adapter-perl cpan-libconstant-perl cpan-libdata-dumper-perl cpan-libdigest-md5-perl
cpan-libexporter-perl cpan-libgetopt-long-perl cpan-libhtml-template-pro-perl cpan-liblocale-codes-perl
cpan-libmime-base64-perl cpan-libmodule-pluggable-perl cpan-libparams-util-perl cpan-libperl-ldap-perl
cpan-libpod-parser-perl cpan-libscalar-list-utils-perl cpan-libsms-send-perl cpan-libtest-manifest-perl
cpan-libtest-simple-perl cpan-libtext-csv-perl cpan-libtext-csv-xs-perl cpan-libtext-tabs-wrap-perl
cpan-libtime-hires-perl cpan-libunicode-normalize-perl cpan-libxml-libxml-perl cpan-libxml-rss-perl
libbarcode-code128-perl libclass-returnvalue-perl libdata-ical-perl libdigest-perl libfile-temp-perl
libgd-barcode-perl libgd-gd2-noxpm-perl libhtml-scrubber-perl libnet-z3950-zoom-perl
libpdf-reuse-barcode-perl libpdf-reuse-perl libschedule-at-perl libtext-vfile-asdata-perl
The following NEW packages will be installed:
cpan-libalgorithm-checkdigits-perl cpan-libansicolor-perl cpan-libbiblio-endnotestyle-perl
cpan-libbundle-kohasupport-perl cpan-libcgi-perl cpan-libclass-adapter-perl cpan-libconstant-perl
cpan-libdata-dumper-perl cpan-libdigest-md5-perl cpan-libexporter-perl cpan-libgetopt-long-perl
cpan-libhtml-template-pro-perl cpan-liblocale-codes-perl cpan-libmime-base64-perl
cpan-libmodule-pluggable-perl cpan-libparams-util-perl cpan-libperl-ldap-perl cpan-libpod-parser-perl
cpan-libscalar-list-utils-perl cpan-libsms-send-perl cpan-libtest-manifest-perl cpan-libtest-simple-perl
cpan-libtext-csv-perl cpan-libtext-csv-xs-perl cpan-libtext-tabs-wrap-perl cpan-libtime-hires-perl
cpan-libunicode-normalize-perl cpan-libxml-libxml-perl cpan-libxml-rss-perl libbarcode-code128-perl
libclass-returnvalue-perl libdata-ical-perl libdigest-perl libfile-temp-perl libgd-barcode-perl
libgd-gd2-noxpm-perl libhtml-scrubber-perl libnet-z3950-zoom-perl libpdf-reuse-barcode-perl libpdf-reuse-perl
libschedule-at-perl libtext-vfile-asdata-perl
0 upgraded, 42 newly installed, 0 to remove and 39 not upgraded.
Need to get 3046kB of archives.
After this operation, 23.6MB of additional disk space will be used.
Do you want to continue [Y/n]?

The trick thing is, it will use the packages in the main debian repo first (like libfile-temp-perl above) and then use the package built from cpan if it can’t find one (like cpan-libtest-manifest-perl)

Ive only built packages for amd64 at this point (well a lot are architecture independent), but if others would find it useful I can add i386 packages to my repo as well.

A geeky way to spend your lunch break

So I spent 20 mins whacking some caching in around the _build_langauges_arrayref subroutine in C4::Languages in Koha.
Before and After. Ill take that as a win. All that needs to be done now is add code to the script that handles updating the language syspref, so that it clears the cache when its changed.

Koha profiling

So far ive only profiled opac-search.pl and opac-main.pl, the results are here. From them I can see doing some caching in the C4::Languages module might speed things up. If you look here you can see its doing a lot of work, and it does this work for each page. So if we cache the data structure resulting from the build_langauges_arrayref, and just return that in the subsequent calls, we should win some decent performance.
Ill work on that tomorrow night, and post the new profiles

Too much weights, not enough speed work

Following on from testing Koha with memcached, I decided to test the opac with mod_perl, mod_expires and mod_deflate.
So with no mod_perl and no caching
time curl http://opac.koha.workbuffer.org/cgi-bin/koha/opac-search.pl?q=a
real 0m2.993s

And with mod_perl
time curl http://opac.koha.workbuffer.org/cgi-bin/koha/opac-search.pl?q=a
real 0m0.657s

And now opac main is down too

real 0m0.010s

This of course isn’t really testing mod_expires or mod_deflate, but certainly telling the browser to cache the images, css and javascript helps out a lot there too.

(Oh and if you haven’t seen the movie Once were Warriors, the title won’t make sense, i’m sure if you search on youtube you can find the clip)

Testing Koha with memcached

Ive been doing some work rewriting some of the scripts in Koha to use memcached where possible.

Heres some loadtesting on opac-main.pl using straight CGI and no caching.


Maximum connect burst length: 1
Total: connections 20 requests 20 replies 20 test-duration 39.896 s
Connection rate: 0.5 conn/s (1994.8 ms/conn, <=20 concurrent connections)
Connection time [ms]: min 35438.7 avg 37343.4 max 39782.2 median 36828.5 stddev 1409.9
Connection time [ms]: connect 0.9
Connection length [replies/conn]: 1.000
Request rate: 0.5 req/s (1994.8 ms/req)
Request size [B]: 64.0
Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (7 samples)
Reply time [ms]: response 37135.8 transfer 206.7
Reply size [B]: header 167.0 content 1156.0 footer 2.0 (total 1325.0)
Reply status: 1xx=0 2xx=20 3xx=0 4xx=0 5xx=0
CPU time [s]: user 3.82 system 15.69 (user 9.6% system 39.3% total 48.9%)
Net I/O: 0.7 KB/s (0.0*10^6 bps)

Heres some with caching switched on

Total: connections 100 requests 94 replies 94 test-duration 24.899 s
Connection rate: 4.0 conn/s (249.0 ms/conn, <=7 concurrent connections)
Connection time [ms]: min 90.9 avg 142.9 max 265.7 median 144.5 stddev 37.1
Connection time [ms]: connect 1.4
Connection length [replies/conn]: 1.000
Request rate: 3.8 req/s (264.9 ms/req)
Request size [B]: 64.0
Reply rate [replies/s]: min 0.0 avg 4.7 max 9.6 stddev 5.4 (4 samples)
Reply time [ms]: response 136.7 transfer 4.9
Reply size [B]: header 186.0 content 5847.0 footer 1.0 (total 6034.0)
Reply status: 1xx=0 2xx=94 3xx=0 4xx=0 5xx=0
CPU time [s]: user 8.53 system 14.22 (user 34.3% system 57.1% total 91.4%)
Net I/O: 22.5 KB/s (0.2*10^6 bps)

So without using the cache we were getting an average of 37343.4 milliseconds to reply. With the cache on that drops to 147.9 … which is a fairly serious saving. This is of course when the machine is under load. If we just run some basic curl tests
Without cache
time curl http://203.97.214.51:8080/
real    0m2.167s

With cache
real 0m0.105s

So that matches up with what we saw with the load testing

We cant of course cache this page if the user is logged in, but thats fairly easy to handle, just check if the user is logged in, if not, use the cache.
In the librarian interface there are significant sections of a lot of pages that change very infrequently where we could win a lot with some caching.
I’ve created a branch and will work in it some more, and also make it publicly available so others can test.

Creating a Programmers manual for Koha 3

Using the neat little module Pod::Manual I whacked up a quick script to combine all the man pages for koha into 1, then output it as docbook (xml not sgml).


#!/usr/bin/perl
use Pod::Manual;
my $manual = Pod::Manual->new({ title => 'C4 Manual'});
my $path = $ARGV[0];
my @chapters = <$path/*>;
foreach $chapter (@chapters){
$chapter =~ s/$path///;
$chapter =~ s/.3pm//;
eval {$manual->add_chapter ($chapter);};
}
print $manual->as_docbook();

I then used the dblatex tools to convert it to pdf.
./manual.pl /path/to/man/files > file.xml
dblatex -tpdf file.xml