Difference between revisions of "Installing MB Slave and Search Server on a New Xubuntu Machine"

From MusicBrainz Wiki
m (adding to "category: how to")
 
(34 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The existing instructions are okay, but because they try and cover all bases they can be more confusing than they need to be. So I thought it would be useful to show exactly what is required for just one common scenerio:ijabz.
+
The existing instructions are okay, but because they try and cover all bases they can be more confusing than they need to be. So I thought it would be useful to show exactly what is required for just one common scenario :ijabz.
  
 
==Install software not available from apt==
 
==Install software not available from apt==
  
- Download ISO image from http://xubuntu.org/ and burn to CD
+
- Download Desktop ISO image from http://xubuntu.org/ (I used Xubuntu 12.10 but I expect later versions should work) and burn to CD
  
- Install Xubunu 12.10 from CD
+
- Install Xbuntu from the CD
  
 
- Download and unzip mbslave from https://bitbucket.org/lalinsky/mbslave
 
- Download and unzip mbslave from https://bitbucket.org/lalinsky/mbslave
  
- Download an unzip psycopg 2-2.4.6 from http://initd.org/psycopg/
+
- Download an unzip psycopg from http://initd.org/psycopg/ (I used pyscogs 2.4.6 but later versions should work)
  
==Download Musicbrainz Datadumps==
+
==Download MusicBrainz Data Dumps==
 +
Download Musicbrainz data dumps from:
 +
* US mirror: http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/
 +
* Danish mirror: http://mirrors.dotsrc.org/MusicBrainz/data/fullexport/
 
<pre>
 
<pre>
 
cp /home/paul/Downloads/mbdump*.bz2 /home/paul/Downloads/lalinsky-mbslave*
 
cp /home/paul/Downloads/mbdump*.bz2 /home/paul/Downloads/lalinsky-mbslave*
Line 24: Line 27:
 
==Configue psycogs2==
 
==Configue psycogs2==
 
<pre>
 
<pre>
sudo apt-get install python-dev
+
sudo apt-get install python python-psycopg2
cd
 
cd Downloads/psycopg2-2.4.6
 
sudo python setup.py install
 
 
</pre>
 
</pre>
  
Line 36: Line 36:
 
cp mbslave.conf.default mbslave.conf
 
cp mbslave.conf.default mbslave.conf
 
</pre>
 
</pre>
uncomment password line and set to musicbrainz
+
Uncomment password line and set to musicbrainz
  
==Setup Musicbrainz Database==
+
==Setup MusicBrainz Database==
 
<pre>
 
<pre>
 
sudo su - postgres
 
sudo su - postgres
 +
</pre>
 +
and enter root password
 +
<pre>
 
createuser musicbrainz
 
createuser musicbrainz
 +
</pre>
 +
Type y when asked if want to set up as superuser
 +
<pre>
 
createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz
 
createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz
 
psql musicbrainz
 
psql musicbrainz
 
ALTER ROLE musicbrainz WITH PASSWORD 'musicbrainz';
 
ALTER ROLE musicbrainz WITH PASSWORD 'musicbrainz';
 
Cntl-D
 
Cntl-D
 +
</pre>
 +
TODO
 +
Configure pg_conf file so that can connect to plsql using musicbrainz from any unix account
 +
 
cd /home/paul/Downloads/lalinsky-mbslave*
 
cd /home/paul/Downloads/lalinsky-mbslave*
 +
Put the following into a shell script, then run:
 +
<pre>
 +
#!/bin/bash
 +
set -xv
 
echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S
 
echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S
 
echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S
 
echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S
Line 57: Line 71:
 
./mbslave-remap-schema.py <sql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/CreateIndexes.sql | grep -vE '(collate|page_index|tracklist_index)' | ./mbslave-psql.py
+
./mbslave-remap-schema.py <sql/CreateIndexes.sql | grep -vE '(collate|page_index|gist)' | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/statistics/CreateIndexes.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/statistics/CreateIndexes.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/caa/CreateIndexes.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/caa/CreateIndexes.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/CreateSimpleViews.sql | ./mbslave-psql.py
 
./mbslave-remap-schema.py <sql/CreateSimpleViews.sql | ./mbslave-psql.py
 +
</pre>
 +
Exit out of postgres user
 +
<pre>
 
Cntl-D
 
Cntl-D
 
</pre>
 
</pre>
Line 66: Line 83:
 
==Build Search Server==  
 
==Build Search Server==  
 
<pre>
 
<pre>
sudo apt-get install subversion openjdk-7-jdk maven2 tomcat7  
+
sudo apt-get install subversion openjdk-7-jdk maven2 tomcat7 git
 
cd
 
cd
 
mkdir indexdata
 
mkdir indexdata
 
mkdir -p code/musicbrainz
 
mkdir -p code/musicbrainz
 
cd code/musicbrainz
 
cd code/musicbrainz
svn co http://svn.musicbrainz.org/mmd-schema/trunk mmd-schema
+
git clone https://github.com/metabrainz/mmd-schema.git mmd-schema
 
svn co http://svn.musicbrainz.org/search_server/trunk search_server
 
svn co http://svn.musicbrainz.org/search_server/trunk search_server
 
cd mmd-schema/brainz-mmd2-jaxb
 
cd mmd-schema/brainz-mmd2-jaxb
Line 79: Line 96:
 
</pre>
 
</pre>
 
Edit web.xml change to <param-value>/home/paul/indexdata</param-value>
 
Edit web.xml change to <param-value>/home/paul/indexdata</param-value>
 +
<pre>
 +
mvn package
 +
</pre>
 +
 +
==Configure postgressql for better performance==
 +
<pre>
 +
sudo vi /etc/postgresql/9.1/main/postgresql.conf
 +
</pre>
 +
- set shared_buffers=800MB
 +
 +
- set work_mem=20MB
 +
 +
- set effective_cache_size = 250MB
 +
 +
<pre>
 +
sudo sysctl -w kernel.shmmax=17179869184
 +
sudo sysctl -w kernel.shmall=17179869184
 +
sudo service postgresql stop
 +
sudo service postgresql start
 +
</pre>
 +
TODO:Add to startup so don't have to redo on every reboot
  
 
==Build Indexes==
 
==Build Indexes==
 
<pre>
 
<pre>
 
cd /home/paul/code/musicbrainz/search_server/index/target
 
cd /home/paul/code/musicbrainz/search_server/index/target
java -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --indexes-dir /home/paul/indexdata -d musicbrainz -u musicbrainz -p musicbrainz
+
java -Xmx500M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --indexes-dir /home/paul/indexdata -d musicbrainz -u musicbrainz -p musicbrainz
 
</pre>
 
</pre>
 +
 +
==Install Jetty server==
 +
<pre>
 +
sudo apt-get install jetty  (note only install old jetty 6, but will do for now)
 +
vi /etc/init.d/jetty (Remove JETTY_STOP line)
 +
</pre>
 +
 +
==Deploy Search Server onto Jetty==
 +
<pre>
 +
cd /usr/share/jetty/webapps
 +
rm -r ROOT
 +
sudo cp /home/paul/code/musicbrainz/search_server/servlet/target/searchserver.war ROOT.war
 +
sudo /etc/init.d/jetty start
 +
</pre>
 +
 +
[[Category:How To]]

Latest revision as of 16:44, 20 May 2016

The existing instructions are okay, but because they try and cover all bases they can be more confusing than they need to be. So I thought it would be useful to show exactly what is required for just one common scenario :ijabz.

Install software not available from apt

- Download Desktop ISO image from http://xubuntu.org/ (I used Xubuntu 12.10 but I expect later versions should work) and burn to CD

- Install Xbuntu from the CD

- Download and unzip mbslave from https://bitbucket.org/lalinsky/mbslave

- Download an unzip psycopg from http://initd.org/psycopg/ (I used pyscogs 2.4.6 but later versions should work)

Download MusicBrainz Data Dumps

Download Musicbrainz data dumps from:

cp /home/paul/Downloads/mbdump*.bz2 /home/paul/Downloads/lalinsky-mbslave*

Install Postgres

sudo apt-get install postgresql-9.1 postgresql-server-dev-9.1 postgresql-contrib-9.1
export PATH=$PATH:/usr/lib/postgresql/9.1/bin

Configue psycogs2

sudo apt-get install python python-psycopg2

Configure mbslave

cd
cd Downloads/lalinsky-mbslave*
cp mbslave.conf.default mbslave.conf

Uncomment password line and set to musicbrainz

Setup MusicBrainz Database

sudo su - postgres

and enter root password

createuser musicbrainz

Type y when asked if want to set up as superuser

createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz
psql musicbrainz
ALTER ROLE musicbrainz WITH PASSWORD 'musicbrainz';
Cntl-D

TODO Configure pg_conf file so that can connect to plsql using musicbrainz from any unix account

cd /home/paul/Downloads/lalinsky-mbslave* Put the following into a shell script, then run:

#!/bin/bash
set -xv
echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S
echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S
echo 'CREATE SCHEMA cover_art_archive;' | ./mbslave-psql.py -S
./mbslave-remap-schema.py <sql/CreateTables.sql | sed 's/CUBE/TEXT/' | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/statistics/CreateTables.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/caa/CreateTables.sql | ./mbslave-psql.py
./mbslave-import.py mbdump.tar.bz2 mbdump-derived.tar.bz2 mbdump-cdstubs.tar.bz2 mbdump-cover-art-archive.tar.bz2
./mbslave-remap-schema.py <sql/CreatePrimaryKeys.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/CreateIndexes.sql | grep -vE '(collate|page_index|gist)' | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/statistics/CreateIndexes.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/caa/CreateIndexes.sql | ./mbslave-psql.py
./mbslave-remap-schema.py <sql/CreateSimpleViews.sql | ./mbslave-psql.py

Exit out of postgres user

Cntl-D

Build Search Server

sudo apt-get install subversion openjdk-7-jdk maven2 tomcat7 git
cd
mkdir indexdata
mkdir -p code/musicbrainz
cd code/musicbrainz
git clone https://github.com/metabrainz/mmd-schema.git mmd-schema
svn co http://svn.musicbrainz.org/search_server/trunk search_server
cd mmd-schema/brainz-mmd2-jaxb
mvn install
cd
cd code/musicbrainz/search_server/servlet/src/main/webapp/WEB-INF

Edit web.xml change to <param-value>/home/paul/indexdata</param-value>

mvn package

Configure postgressql for better performance

sudo vi /etc/postgresql/9.1/main/postgresql.conf

- set shared_buffers=800MB

- set work_mem=20MB

- set effective_cache_size = 250MB

sudo sysctl -w kernel.shmmax=17179869184
sudo sysctl -w kernel.shmall=17179869184
sudo service postgresql stop
sudo service postgresql start

TODO:Add to startup so don't have to redo on every reboot

Build Indexes

cd /home/paul/code/musicbrainz/search_server/index/target
java -Xmx500M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --indexes-dir /home/paul/indexdata -d musicbrainz -u musicbrainz -p musicbrainz

Install Jetty server

sudo apt-get install jetty  (note only install old jetty 6, but will do for now)
vi /etc/init.d/jetty (Remove JETTY_STOP line)

Deploy Search Server onto Jetty

cd /usr/share/jetty/webapps
rm -r ROOT
sudo cp /home/paul/code/musicbrainz/search_server/servlet/target/searchserver.war ROOT.war
sudo /etc/init.d/jetty start