DISCLAIMER
This document explains how to setup a local copy of WikiPathways. This is not for the faint of heart!
The following documentation is mainly targeting WikiPathways developers interested in setting up a local development
Please do not attempt this on a deadline. Please do not attempt this with any expectation of success without a furrowed brow,
frustration, unexplained phenomena, and a few curse words. The documentation and public mailing lists are still maturing,
and WikiPathways admins are busy developing WikiPathways. You have been warned!
instance, purely for hacking or demos. Developers may also refer to this to setup authorized mirrors of WikiPathways.
Theoretically, one could also install a local instance as an internal wiki for a unique and dedicated purpose.
But, if you are simply looking to support private or proprietary pathways, the main site at wikipathways.org can do this.
You do not need a local instance.
System Requirements
The installation should be done on Ubuntu. Installation on RedHat, Fedora, CentOS and Windows Vista have failed or are incomplete.
- Ubuntu 8.04.3+ (32 or 64 bit)
- min 40G free space
- min 2G RAM
- Java 5+
Dependencies
Install these first:
- mysql
- apache2
- java sdk (>= 5)
- ant
- inkscape
- php5 (version 5.2.x)
- svn
- php-xml (needed on redhat)
- wso2-wsf-php 2.0.0 (needed for SOAP web service only)
- pecl_http php module (needed for SOAP web service only)
On a ubuntu system, you can simply apt-get:
mysql-server ant sun-java5-sdk apache2 inkscape php5
Make sure Ubuntu uses the right version of Java! On ubuntu the default java is set to /usr/lib/jvm/java-gcj/jre/bin/java, even if you installed sun-java5-sdk. To change this, run:
update-alternatives --config java #Now choose /usr/lib/jvm/java-6-sun/jre/bin/java update-alternatives --config javac #Now choose /usr/lib/jvm/java-6-sun/bin/javac
Wso2-wsf-php is not in the Ubuntu repositories. Please follow these instructions to build the library from source. Make sure you have the packages libxml2 and libxml++2.6-dev installed!
The pecl_http php module is not in the Ubuntu repositories, but can be installed using 'pear'. Note that there is a dependency on libcurl4-gnutls-dev. On Ubuntu, you can install pecl_http like this:
sudo apt-get install libcurl4-gnutls-dev pear config-set php_ini /etc/php5/apache2/php.ini sudo pecl install pecl_http sudo apache2ctl restart
On redhat, you can 'yum' some of these, others you will have to install via rpm, e.g.:
>rpm -ivh inkscape-0.44-0.i386.rpm (may have dependencies) -->rpm -ivh libglitz1-0.5.6-1mdv2007.0.i586.rpm -->yum install gtkspell >rpm -ivh subversion-1.4.6-1.i386.rpm
Checkout the website from svn
Checkout the website and php scripts from the svn server and place them in your site directory (e.g. /var/www/wikipathways).
The best way to do this is to, first, make the directory '/var/www/wikipathways/', 'cd' into the directory, and then check out the code directly into that directory. You may want to checkout the latest 'tag' rather than the 'trunk'. Tags are the official releases of WikiPathways and MediaWiki updates. For example:
mkdir /var/www/wikipathways cd /var/www/wikipathways svn co http://svn.bigcat.unimaas.nl/wikipathways/tags/milestone_15.0/ . (for milestone 15, and don't forget the '.' at the end)
Then, simply swtich to the latest tag after each milestone release:
svn switch http://svn.bigcat.unimaas.nl/wikipathways/tags/milestone_16.0/ (for milestone 16)
This strategy will keep you code up-to-date with our changes and MediaWiki changes, while maintaining any custom changes you may choose to make.
Create a pass.php file
Create a file named pass.php in the directory in which you checked out the wikipathways code. This file should contain:
<?php $wgDBname = "wikidb"; #Replace with name of your local wikipathways database $wgDBuser = "wikiuser"; #Replace with name of your local wikipathways database user $wgDBpassword = "password"; #Replace with password of you local wikipathways database user $wgScriptPath = "/wikipathways-local"; #Replace with the path of the wikipathways directory relative to the base url (see http://www.mediawiki.org/wiki/Manual:%24wgScriptPath) //Disable email on local setup $wgEnableEmail = false; $wgEnableUserEmail = false; $wgEnotifUserTalk = false; # UPO $wgEnotifWatchlist = false; # UPO //ReCaptcha Keys //These are unique per server. You can generate a set of keys //for your server at http://recaptcha.net/api/getKey $recaptcha_public_key = '******'; $recaptcha_private_key = '******'; # Replace xxxxxxx-x with YOUR GoogleAnalytics UA number $wgGoogleAnalyticsAccount = "UA-xxxxxxx-x"; ?>
Modify the list of supported species via pass.php
You can add/remove support for species on your local wikipathways installation. To do this, you can use the static methods in the class Organism. For example, to remove support for 'Homo sapiens' and add support for 'Carinotetraodon travancoricus', add these lines to pass.php:
//Remove support for H. Sapiens
Organism::remove(Organism::getByLatinName('Homo sapiens');
//Add support for C. travancoricus
Organism::register('Carinotetraodon travancoricus', Ct);
Create a cache directory
mkdir /var/www/wikipathways/wpi/cache
Create a wiki database user
Create a user for your local wikipathways database:
create database wikidb; grant create, select, insert, update, delete, alter, lock tables, show view, create view, drop on wikidb.* to 'wikiuser'@'localhost' identified by 'password'; flush privileges; \q
Where 'wikidb' can be replaced with the name you want to use for the database. 'wikiuser' and 'password' can be replaced by a custom username and password. Also see here.
Copy wiki database contents
Choose one of the following two options to create the database contents.
Start with clean wiki database
Use this step if you don't have ssh access to the WikiPathways server or if you want to start without any content (pathways/users).
- Download and extract the following sql dump: http://www.bigcat.unimaas.nl/wikipathways_clean.sql.gz
- Import this dump in your local mysql database (replace wikidb with the database name you chose in the previous step):
use wikidb; source wikipathways_clean.sql;
Optionally, you can copy content from www.wikipathways.org to your local installation manually after the site is setup.
The clean database dump contains two user accounts that are used for maintenance:
- MaintBot
- WikiSysop
The initial password for both accounts is "0000". Please change this after installing the database!
Mirror WikiPathways content
Use this step if you have ssh access to the WikiPathways server and you want to mirror the current contents (including users and pathways). To include the WikiPathways content in you local copy:
- Create a database dump:
mysqldump -v -u wikipathwaysuser -p wikipathways > wikipathways.sql
- Import this in your local mysql database (replace wikidb with the database name you chose in the previous step):
use wikidb; source wikipathways.sql;
- Note: if you are setting up a development site, please scrub the user table clean of private data using the following SQL statement:
#Remove private user data and email notification settings from user table: UPDATE user SET user_name = user_id, user_real_name='Test', user_password='0', user_newpassword='0', user_email='test@example.org', user_options='quickbar=1\nunderline=2\ncols=80\nrows=25\nsearchlimit=20\ncontextlines=5\ncontextchars=50\nskin=\nmath=1\nrcdays=7\nrclimit=50\nwllimit=250\nhighlightbroken=1\nstubthreshold=0\npreviewontop=1\neditsection=1\neditsectiononrightclick=0\nshowtoc=1\nshowtoolbar=1\ndate=default\nimagesize=2\nthumbsize=2\nrememberpassword=0\nenotifwatchlistpages=0\nenotifusertalkpages=0\nenotifminoredits=0\nenotifrevealaddr=0\nshownumberswatching=1\nfancysig=0\nexternaleditor=0\nexternaldiff=0\nshowjumplinks=1\nnumberheadings=0\nuselivepreview=0\nwatchlistdays=3\nvariant=en\nlanguage=en\nsearchNs0=1\nsearchNs102=1\nsearchNs103=1\nsearchNs100=', user_token='', user_email_token='' WHERE user_name != 'WikiSysop';
- Copy the images directory (/var/www/wikipathways/images) to your local installation.
- Also take note of cron jobs: crontab -e
Set directory permissions
Make sure the following directories exist and the apache user (www-data) has read/write access (775):
- /var/www/wikipathways/images
- /var/www/wikipathways/wpi/tmp
- /var/www/wikipathways/wpi/cache
- /var/www/wikipathways/wpi/applet
Create the applet
Check out the pathvisio source from the svn server
Run
export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun/ (or your path to java 5) ant install-wpi -Dwikipathways.dest=/path/to/wikipathways -Dkeystore.password=****
Where you replace /path/to/wikipathways with the right directory (/var/www/wikipathways by default) and provide the right keystore password. Note that this ant task will need to sign the applet jars with a valid certificate. It will use the keystore lib/visioKeyStore in the PathVisio svn repository. You may overwrite this file with your own keystore and set the correct keystore password in the ant variable "keystore.password". For instructions on how to create your own keystore, see here.
Setup site statistics
The statistics script that calculates the number of pathways and genes should be updated using a cron job. Add this line to your crontab file to update the statistics once a day:
00 00 * * * cd /var/www/wikipathways/wpi && php StatisticsCache.php
Setup linkouts
To maintain up-to-date linkouts to external databases (Ensembl, Entrez Gene, HMDB etc), a file needs to be downloaded periodically from bridgedb.org. You can add this to a cronjob. Run "crontab -e" and add this line to update the file every day at midnight:
00 00 * * * cd /var/www/wikipathways/wpi && php DataSourcesCache.php
Trigger weekly flatfile generation for downloads
Add a script to your cron.weekly dir with update-overrides to generate flatfiles for the download page. Repeat for all species. Also note repeats for 'tab' and 'html' versions and for 'mapped' and 'original' versions:
wget -q --delete-after http://www.wikipathways.org/wpi/pathway_content_flatfile.php?species=Homo%20sapiens\&output=tab\&update=yes wget -q --delete-after http://www.wikipathways.org/wpi/pathway_content_flatfile.php?species=Homo%20sapiens\&output=tab\&mapping=off\&update=yes wget -q --delete-after http://www.wikipathways.org/wpi/pathway_content_flatfile.php?species=Homo%20sapiens\&output=html\&update=yes wget -q --delete-after http://www.wikipathways.org/wpi/pathway_content_flatfile.php?species=Homo%20sapiens\&output=html\&mapping=off\&update=yes
Setup the synonym database server
The synonym databases are needed to link genes, proteins and metabolites to external databases (such as Ensembl). In this step you will setup an Apache Derby server to host these databases.
- Create a database directory, e.g., /home/wikipathways/database/
- Download and install Apache Derby into the database directory.
- Download the PathVisio synonym databases (.bridge files) here. Note: The .zip files are compressed archives of .bridge files, which are themselves uncompressed archives of Derby database files. So, you need to unzip the .zip, then unzip the .bridge in order to get to the Derby. The next step works with .bridge files directly, handling the second unzip for you.
- Download the following helper scripts into the same directory as the .bridge files:
- Run LinkDatabase.pl in order to automatically (or interactively) create the necessary symlinks to the databases so that they can be used by WikiPathways. The script extracts the .bridge archives, renames the derby database folders and creates symlinks using the conventional genus species names. In the case of metabolite databases, it creates a symlink using the word 'metabolites'. It also cleans up old links and database folders.
- Alternatively, (if you have access) copy the whole database directory "/home/wikipathways/database" from the WikiPathways server.
- Now make the derby server run automatically. Create this script in /usr/sbin/myderbydaemon:
#!/bin/sh export DERBY_INSTALL=/home/wikipathways/database/db-derby-10.4.2.0-bin/ export DERBY_HOME=/home/wikipathways/database/ export JAVA_HOME=/usr/lib/jvm/java-6-sun/ export CLASSPATH=$DERBY_INSTALL/lib/derby.jar:$DERBY_INSTALL/lib/derbytools.jar:$DERBY_INSTALL/lib/derbynet.jar:. java -Dderby.system.home=$DERBY_HOME org.apache.derby.drda.NetworkServerControl start -h 0.0.0.0
On Debian and Ubuntu:
- sudo cp /etc/init.d/skeleton /etc/init.d/myderbydaemon
- then, apply the following patch. Note: the patch does 3 things (which you can simply do yourself if you're not comfortable with patches):
- insert #!/bin/sh line
- replace Author information (feel free to credit the actual author of the script, Martijn van Iersel)
- replace Name of daemon with "myderbydaemon"
manager@bigcat0025uns40mw:~$ diff /etc/init.d/skeleton /etc/init.d/myderbydaemon 0a1 > #!/bin/sh 13c14 < # Author: Foo Bar <foobar@baz.org> --- > # Author: Martijn van Iersel <martijn.vaniersel@bigcat.unimaas.nl> 23c24 < NAME=daemonexecutablename --- > NAME=myderbydaemon 55a57 >
- now, create some soft links:
sudo ln -s /etc/init.d/myderbydaemon /etc/rc2.d/S92myderbydaemon sudo ln -s /etc/init.d/myderbydaemon /etc/rc3.d/S92myderbydaemon sudo ln -s /etc/init.d/myderbydaemon /etc/rc4.d/S92myderbydaemon sudo ln -s /etc/init.d/myderbydaemon /etc/rc5.d/S92myderbydaemon
On Redhat:
- copy template init script from vi /usr/share/doc/initscripts-<package>/sysvinitfiles
- fill-out (or comment out) all <tagged text>
- save to /etc/init.d/myderbydaemon
- create symbolic links (above)
Firewall Notes:
- Make sure you punch a hole through port 1527 for Derby connections
Setup bots (optional)
We created several scripts ('bots') that check for inconsistencies in the pathways on the wiki. These bots work through the web service, so they can run on another server (recommended).
In the instructions below, we assume the bots are placed in the directory '/home/wikipathways/bots/wikipathways'.
To build the bots, go to the directory with your PathVisio svn working copy and run:
ant tarbin-wpclient -Dwsdl.url=http://www.wikipathways.org/wpi/webservice/webservice.php?wsdl cp wikipathways_client_bin-1.1-rXXXX.tar.gz /home/wikipathways/bots/wikipathways/ cd ~/bots/wikipathways tar -xf wikipathways_client_bin-1.1-rXXXX.tar.gz --strip=1
Each bot requires a configuration file. Here are the configuration files we use for www.wikipathways.org:
- Connector bot:
webservice-url=http://www.wikipathways.org/wpi/webservice/webservice.php cache-path=/home/manager/wikipathways-bots/wikipathways/cache user=MaintBot pass=***** threshold=80
- Xref bot:
webservice-url=http://www.wikipathways.org/wpi/webservice/webservice.php cache-path=/home/manager/wikipathways-bots/wikipathways/cache user=MaintBot pass=***** threshold=90 gdb-config=/home/manager/gdb.config
* Description bot:
webservice-url=http://www.wikipathways.org/wpi/webservice/webservice.php cache-path=/home/manager/wikipathways-bots/wikipathways/cache user=MaintBot pass=*****
In order to run the Xref bot, you will need a synonym database configuration file (gdb-config). See the instructions on the Lucene indexer (below) for instructions on how to create such a file.
You can now create a script for each bot to run it, e.g.:
#!/bin/sh DATE_TIME=`date +%y%m%d%k%M%S` #Description bot REPORT_FILE=report-$DATE_TIME-description java -cp wikipathways-client.jar org.pathvisio.wikipathways.bots.DescriptionBot descr-bot.props $REPORT_FILE chmod a+r $REPORT_FILE.html scp $REPORT_FILE.html wikipathways@wikipathways.org:/home/wikipathways/bots/wikipathways/report-description.html:
The symbolic links to the html report files can be used on the WikiPathways? website to link to the most recent bot report. You can run the scripts periodically using a cron job.
Setup Lucene indexer (optional, but used by SOAP web service)
You can optionally setup a lucene indexer service that is needed for the search functions in the webservice. The lucene indexer is implemented as a java tool that builds a lucene index, updates it at regular intervals and provides a REST service to query the index. You can setup the indexer service next to your wikipathways installation on the same server, or on a dedicated server.
1. Build the lucene indexer jar
Make sure the wikipathways-client has been built. In the pathvisio code directory, run:
ant jar-wpclient -Dwsdl.url=http://www.wikipathways.org/wpi/webservice/webservice.php?wsdl
Now build the Lucene indexer service:
cd tools/lucene-indexer ant dist
2. Start the indexer service
You can now copy the contents of the created dist/ directory to the location you want to run the service from, in this example it's a different directory on the same server:
cp -R dist/ /home/wikipathways/wikipathways-index-service
The indexer needs two configuration files to work:
1. bridge.config: This file points the indexer to the available synonym databases for each organism. The file contains two tab separated columns, where the first column is a list of species and the second column the corresponding .pgdb files. If a synonym database applies to all species, use '*' in the first column. Example:
Homo sapiens /home/wikipathways/database/Hs_Derby_20081119.pgdb Mus musculus /home/wikipathways/database/Mm_Derby_20081119.pgdb Rattus norvegicus /home/wikipathways/database/Rn_Derby_20081119.pgdb Danio rerio /home/wikipathways/database/Dr_Derby_20081119.pgdb Caenorhabditis elegans /home/wikipathways/database/Ce_Derby_20081205.pgdb Saccharomyces cerevisiae /home/wikipathways/database/Sc_Derby_20081212.pgdb Drosophila melanogaster /home/wikipathways/database/Dm_Derby_20081119.pgdb Oryza japonica /home/wikipathways/database/Oj_Derby_20081109 Arabidopsis thaliana /home/wikipathways/database/At_Derby_20081103 * /home/wikipathways/database/metabolites_080916.pgdb
2. indexer.config: This file contains several parameters for the indexer:
- rpc_url: The url to the SOAP web service
- update_interval: The interval on which to check for updates (in milliseconds)
- index_path: The path to store the index files
- cache_path: The path to store the cached pathway files
Example:
rpc_url=http://www.wikipathways.org/wpi/webservice/webservice.php update_interval=60000 index_path=/var/local/wikipathways/index cache_path=/var/local/wikipathways/cache
Now you can run the indexer service using the following shell script, which will also log any console output to a log file:
JAVA_HOME="/usr/lib/jvm/java-6-sun" JAVA_EXEC="$JAVA_HOME/bin/java" WIKIPATHWAYS_HOME="/home/wikipathways" INDEXER_DIR="$WIKIPATHWAYS_HOME/wikipathways-index-service" PORT="8184" LOG_FILE="$INDEXER_DIR/indexer.log" ARGS="-jar $INDEXER_DIR/dist/wikipathways-indexer-service.jar -b $WIKIPATHWAYS_HOME/gdb.config -i $INDEXER_DIR/indexer.config -p $PORT" eval "$JAVA_EXEC $ARGS" >$LOG_FILE 2>&1
This will start the indexer service on port 8184. The indexer will keep running and updating the index at the interval specified in the configuration file. To prevent the indexer process to be killed after you log out, you can use the 'screen' command:
screen ./run.sh #The previously shown shell script
When you login again, you can resume the screen with:
screen -r
Note:
After updating WikiPathways to a new release, you might need to rebuild and restart the indexer. To make sure the index will be up-to-date, remove the files in index_path before you restart the indexer.
Update:
To make the indexer automatically start after reboot of the server, you can create a daemon script and copy it to /etc/init.d (see below) and run:
sudo update-rc.d indexer-daemon start 51 S .
This is the script currently used for the indexer server on wikipathways.org:
#! /bin/sh
#Original /etc/init.d/skeleton modified for http://mydebian.blogdns.org
# Indexer service config
WIKIPATHWAYS_HOME="/home/wikipathways"
INDEXER_DIR="$WIKIPATHWAYS_HOME/wikipathways-index-service"
PORT="8183"
ARGS="-jar $INDEXER_DIR/dist/wikipathways-indexer-service.jar -b $WIKIPATHWAYS_HOME/gdb.config -i $INDEXER_DIR/indexer.config -p $PORT"
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="WikiPathways indexer service"
NAME="indexer-daemon"
DAEMON=/usr/bin/java
DAEMON_ARGS="$ARGS"
DAEMON_PATH=/home/wikipathways/
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/indexer-daemon
#the user that will run the script
USER=wikipathways
# NO NEED TO MODIFY THE LINES BELOW
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
#
# Function that starts the daemon/service
#
do_start()
{
start-stop-daemon -b -d $DAEMON_PATH --start --quiet --chuid $USER -m -p $PIDFILE --exec $DAEMON -- $DAEMON_ARGS \
|| return 2
}
#
# Function that stops the daemon/service
#
do_stop()
{
start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
RETVAL="$?"
rm -f $PIDFILE
return "$RETVAL"
}
case "$1" in
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
restart)
#
# If the "reload" option is implemented then remove the
# 'force-reload' alias
#
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*)
# Failed to stop
log_end_msg 1
;;
esac
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|restart}" >&2
exit 3
;;
esac
:
3. Configure the webservice
To enable the search functions in the WikiPathways? web service, you need to specify the url to the lucene indexer service. In our example above, the server runs on the same server as the WikiPathways? installation on port 8184. Specify the location in the pass.php file in the wikipathways installation directory (/var/www/wikipathways), by adding the following variable:
$indexServiceUrl = "http://localhost:8184/";
If you run the lucene indexer service on another server, you need to change the url and port accordingly.
Notes
- The google search always redirects to the global (wikipathways) search (so it doesn't search your local copy)! In order to make google search work for you, you will have to establish your own Google CSE account and replace the information in includes/DefaultSettings.php and /wpi/extensions/googleSearch.php.
- If you don't see any pathway images, check the error logs for inkscape errors. If you see this error:
(inkscape:3971): libgnomevfs-WARNING **: Unable to create ~/.gnome2 directory: Permissi on denied
Create directory /var/www/.gnome2 and make www-data the owner:
sudo mkdir /var/www/.gnome2 sudo chown www-data:www-data /var/www/.gnome2/
- If you started with a "clean wiki database", the following pages and images will need to be copied from the main site to support certain optional functions:
Template: PathwayPage:Top TodaysPathway CurationTag Curation:FeaturedPathway UserPage TalkPage Pages: Help pages Footer pages: about, licenses, disclaimers, etc Image files: Curation tag icons Home page icons
- Confirm the following memory settings in LocalSettings.php (or pass.php):
ini_set( 'memory_limit', '128M' ); $wgMaxShellMemory = 512 * 1024;
