Mwlib

From Wikitech
Jump to navigation Jump to search
This page may be outdated or contain incorrect details. Please update it if you can.

Notes

Notes on building mwlib 'offline collection generator' hosts . . .

ocg1 installed - testing cherrypicked-to-production puppet work here
ocg2 installed and are awaiting puppetization
ocg3 has a firmware issue that needs resolution

http://mwlib.readthedocs.org/en/latest/renderserver.html
http://mwlib.readthedocs.org/en/latest/installation.html#ubuntu-install
http://hynek.me/articles/python-deployment-anti-patterns/

Ref. http://wiki.debian.org/Python

  • use site-packages?
  • use dh_python2?

Should these be isolated from stock python packages, i.e. in an alternate library path, etc?
dpkg-buildpackage -rfakeroot -b

Labs pediapress project

bob - build machine, ip 10.4.0.90
pediapress-ocg1 - ubuntu precise test host
pediapress-ocg2 - ubuntu lucid test host
pediapress-packager - ubuntu lucid host for packaging

To Be Determined

1) daemonizing: runit, init, upstart?

2) package individual modules or entire homedir heap as a single package?

3) depending on #2, install modules system-wide or under a user?

4) other user and filesystem layout specifics not covered in 1-3

Build process

puppet does this already

 apt-get install -y dvipng g++ gcc git-core imagemagick libfreetype6-dev libjpeg-dev liblcms-dev \
    libxml2-dev libxslt-dev libz-dev make ocaml-nox pdftk ploticus python python-dev python-docutils \
    python-greenlet python-imaging python-lxml python-pyparsing python-pypdf python-virtualenv \
    texlive-latex-recommended
 groupadd pp
 useradd -m -c "pediapress" -g pp -d /opt/pp pp

this part needs puppetification

 su - pp
 virtualenv --distribute ~/local; export PATH=~/local/bin:$PATH; hash -r; export PIP_INDEX_URL=http://pypi.pediapress.com/simple/
 pip install pil       # provided by python-imaging
 pip install pyfribidi # no additional dependencies
 pip install mwlib     # pulls odfpy-0.9.4, gevent-1.0b2, timelib-0.2.3, bottle-0.10.9, apipkg-1.0, qserve-0.2.7, roman-1.4.0, py-1.4.7, sqlite3dbm-0.1.4
 pip install mwlib.rl  # pulls mwlib.ext, pygments-1.5
 pip install mwlib.zim # pulls pyzim
 set up daemons

Packages to Build

Stock Packages That Should Work

apipkg-1.0
 ubuntu: python-apipkg (lucid:n/a precise:1.0-1.1build1)
bottle-0.10.9
 ubuntu: python-bottle (lucid:0.6.4-1 precise:0.10.6-1)
 source package: git://git.debian.org/collab-maint/python-bottle.git
 http://bottlepy.org/docs/0.10/changelog.html
 stock package looks tricky to backport due to debian/control build dependencies
pygments-1.5
 ubuntu: python-pygments (lucid:1.2.2+dfsg-1ubuntu1 precise:1.4+dfsg-2)
 http://pygments.org/docs/changelog/#version-1-5
 source: http://pypi.python.org/packages/source/P/Pygments/Pygments-1.5.tar.gz
 on precise, it's pulled in as a python-docutils dependency
c-ares
 ubuntu: libc-ares-dev (lucid:1.7.0-1-1 precise:1.7.5-1)
 ubuntu: libc-ares2 (lucid:1.7.0-1 precise:1.7.5-1)
libev
 ubuntu: libev-dev (lucid:1:3.8-1 precise:1:4.11-1)
 ubuntu: libev3 (lucid:1:3.8-1)
 ubuntu: libev4 (precise:1:4.11-1)
reportlab
 ubuntu: python-reportlab (lucid:2.4-1 precise:2.5-1.1build1)
reportlab-accel
 ubuntu: python-reportlab-accel (lucid:2.4-1 precise:2.5-1.1build1)
cython
 ubuntu: cython (lucid:0.11.2-1 precise:0.15.1-1ubuntu1)
python-lxml
 ubuntu: python-lxml (lucid:2.2.4-1 precise:2.3.2-1)
cxxtools
 ubuntu: libcxxtools-dev (lucid:1.4.8-3 precise:2.0-4ubuntu2)
tntdb
 ubuntu: libtntdb-dev (lucid:1.0.1-3build1 precise:1.1-1)
xz
 ubuntu: (?) xz-utils (lucid:4.999.9beta+20091116-1 precise:5.1.1alpha+20110809-3)

New Packages That Build Easily

odfpy-0.9.4
 * built for lucid & precise
py-1.4.7
 * built for lucid & precise
qserve-0.2.7
 * built for lucid & precise
sqlite3dbm-0.1.4
 * built for lucid & precise
pyfribidi-0.11.0
 ubuntu: python-pyfribidi (lucid:0.10.0-2 precise:0.10.0-4), both patched for buffer overflow
 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=663189
 http://pypi.python.org/pypi/pyfribidi/
 upstream appears dead, pediapress released 0.11.0 but changelog ends at 0.10.1
 comment: 0.11 and 0.10.1 have the same API. 0.10.0 won't work and is still broken on ubuntu precise
 notes: remove fribidi-src before packaging
 * built for lucid & precise but needs to either replace stock package or be packaged with a different name

New Packages That Require Significant Refactoring Before Build

pyzim (& zimlib, zimwriter)
 source: http://www.openzim.org/Releases
 http://pypi.python.org/pypi/pyzim
 build requires: cxxtools =2.0, tntdb =1.1 (for zimwriter), tntnet =2.0 (for zimreader), libxz-dev, autoconf, automake, libtool
 uses a Makefile to build cxxtools, tntdb, xz, zimlib and zimwriter so you'll want remove the vendor directory
 recommend:
 - fulfill requires from stock cxxtools, tntdb, xz
 - build zimlib, pyzim, zimwriter as necessary as discrete packages
gevent-1.0b2
 backporting 1.0b2 to fix bugs Ralf patched and sent upstream in previous versions 
 ubuntu: python-gevent (lucid:n/a precise:0.13.6-1ubuntu1)
 stable: gevent-0.13.7
 beta: gevent-1.0b2 requires greenlet, libev & c-ares, release tarball includes dependencies
 significant changes between 0.13.7 and 1.0b2
 remove included libev and c-ares
 http://packages.debian.org/experimental/python/python-gevent
 recommend:
 - fulfill requires from stock packages
 - create python-pediapress-gevent from 1.0b2 source


mwlib-0.13.6
 "mwlib itself also includes a slightly patched apipkg (v1.1) module. I think I already
 got the changes merged upstream. mwlib also includes a patched EasyTimeline.pl script."
 recommend:
 - determine whether stock apipkg will suffice
 - otherwise backport as a replacement for stock mwlib  
 - possibly package EasyTimeline.pl separately
mwlib.ext-0.12.4
 provides external dependencies needed by the mwlib library
 essentially reportlab 2.5 with some patches
 http://pypi.python.org/pypi/mwlib.ext/0.12.4 (no Makefile)
 - determine whether stock reportlab will suffice
 - otherwise backport as a replacement for stock reportlab 
mwlib.rl-0.12.10
 check with Pediapress re. font install
mwlib.zim-0.2.0
 requires cython, lxml, mwlib, pyzim (pyzim requires zimlib, zimwriter)
 http://pypi.python.org/packages/source/m/mwlib.zim/mwlib.zim-0.2.0.zip (no Makefile)
 recommend:
 - fulfill requirements with stock packages
 - build mwlib.zim itself as new package
timelib-0.2.3
 ./ext-date-lib is an import of php-5.2.10/ext/date/lib for use in static linking
 "timelib includes unmodified sources from php. you should also break those out
 and link against libphp (if there such a thing in ubuntu)."
 * first attempt package failed, won't build as fakeroot
 recommend:
 - preferred: reimplement in python
 - good: rewrite dynamically linked against stock php
 - acceptable: rewrite statically linked against stock php
texvc
 - pediapress way
   git clone https://github.com/pediapress/texvc
   cd texvc; make; make install PREFIX=~/local
 - mediawiki way (?)
   http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/Math/math
   version #? use mediawiki version #? what?
   is debian/copyright sane?
   is version sane for pediapress folks?
 recommend: consult mw and pediapress devs for instructions on what to package