History:Installing MB Slave and Search Server on a New Xubuntu Machine: Difference between revisions
No edit summary |
Reosarevok (talk | contribs) No edit summary |
||
(33 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
{{Status|'''This Page is Glorious History!''' |
|||
This refers to very old MusicBrainz infrastructure including an older, barely related version of the search server. |
|||
}} |
|||
The existing instructions are okay, but because they try and cover all bases they can be more confusing than they need to be. So I thought it would be useful to show exactly what is required for just one common scenario :ijabz. |
The existing instructions are okay, but because they try and cover all bases they can be more confusing than they need to be. So I thought it would be useful to show exactly what is required for just one common scenario :ijabz. |
||
==Install software not available from apt== |
==Install software not available from apt== |
||
- Download ISO image from http://xubuntu.org/ and burn to CD |
- Download Desktop ISO image from http://xubuntu.org/ (I used Xubuntu 12.10 but I expect later versions should work) and burn to CD |
||
- Install |
- Install Xbuntu from the CD |
||
- Download and unzip mbslave from https://bitbucket.org/lalinsky/mbslave |
- Download and unzip mbslave from https://bitbucket.org/lalinsky/mbslave |
||
- Download an unzip psycopg |
- Download an unzip psycopg from http://initd.org/psycopg/ (I used pyscogs 2.4.6 but later versions should work) |
||
==Download MusicBrainz Data Dumps== |
==Download MusicBrainz Data Dumps== |
||
Download Musicbrainz data dumps from: |
|||
* US mirror: http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/ |
|||
* Danish mirror: http://mirrors.dotsrc.org/MusicBrainz/data/fullexport/ |
|||
<pre> |
<pre> |
||
cp /home/paul/Downloads/mbdump*.bz2 /home/paul/Downloads/lalinsky-mbslave* |
cp /home/paul/Downloads/mbdump*.bz2 /home/paul/Downloads/lalinsky-mbslave* |
||
Line 24: | Line 33: | ||
==Configue psycogs2== |
==Configue psycogs2== |
||
<pre> |
<pre> |
||
sudo apt-get install python- |
sudo apt-get install python python-psycopg2 |
||
cd |
|||
cd Downloads/psycopg2-2.4.6 |
|||
sudo python setup.py install |
|||
</pre> |
</pre> |
||
Line 36: | Line 42: | ||
cp mbslave.conf.default mbslave.conf |
cp mbslave.conf.default mbslave.conf |
||
</pre> |
</pre> |
||
Uncomment password line and set to musicbrainz |
|||
==Setup MusicBrainz Database== |
==Setup MusicBrainz Database== |
||
<pre> |
<pre> |
||
sudo su - postgres |
sudo su - postgres |
||
</pre> |
|||
and enter root password |
|||
<pre> |
|||
createuser musicbrainz |
createuser musicbrainz |
||
</pre> |
|||
Type y when asked if want to set up as superuser |
|||
<pre> |
|||
createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz |
createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz |
||
psql musicbrainz |
psql musicbrainz |
||
ALTER ROLE musicbrainz WITH PASSWORD 'musicbrainz'; |
ALTER ROLE musicbrainz WITH PASSWORD 'musicbrainz'; |
||
Cntl-D |
Cntl-D |
||
</pre> |
|||
TODO |
|||
Configure pg_conf file so that can connect to plsql using musicbrainz from any unix account |
|||
cd /home/paul/Downloads/lalinsky-mbslave* |
cd /home/paul/Downloads/lalinsky-mbslave* |
||
Put the following into a shell script, then run: |
|||
<pre> |
|||
#!/bin/bash |
|||
set -xv |
|||
echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S |
echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S |
||
echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S |
echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S |
||
Line 57: | Line 77: | ||
./mbslave-remap-schema.py <sql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py |
./mbslave-remap-schema.py <sql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py |
||
./mbslave-remap-schema.py <sql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py |
./mbslave-remap-schema.py <sql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py |
||
./mbslave-remap-schema.py <sql/CreateIndexes.sql | grep -vE '(collate|page_index| |
./mbslave-remap-schema.py <sql/CreateIndexes.sql | grep -vE '(collate|page_index|gist)' | ./mbslave-psql.py |
||
./mbslave-remap-schema.py <sql/statistics/CreateIndexes.sql | ./mbslave-psql.py |
./mbslave-remap-schema.py <sql/statistics/CreateIndexes.sql | ./mbslave-psql.py |
||
./mbslave-remap-schema.py <sql/caa/CreateIndexes.sql | ./mbslave-psql.py |
./mbslave-remap-schema.py <sql/caa/CreateIndexes.sql | ./mbslave-psql.py |
||
./mbslave-remap-schema.py <sql/CreateSimpleViews.sql | ./mbslave-psql.py |
./mbslave-remap-schema.py <sql/CreateSimpleViews.sql | ./mbslave-psql.py |
||
</pre> |
|||
Exit out of postgres user |
|||
<pre> |
|||
Cntl-D |
Cntl-D |
||
</pre> |
</pre> |
||
Line 66: | Line 89: | ||
==Build Search Server== |
==Build Search Server== |
||
<pre> |
<pre> |
||
sudo apt-get install subversion openjdk-7-jdk maven2 tomcat7 |
sudo apt-get install subversion openjdk-7-jdk maven2 tomcat7 git |
||
cd |
cd |
||
mkdir indexdata |
mkdir indexdata |
||
mkdir -p code/musicbrainz |
mkdir -p code/musicbrainz |
||
cd code/musicbrainz |
cd code/musicbrainz |
||
git clone https://github.com/metabrainz/mmd-schema.git mmd-schema |
|||
svn co http://svn.musicbrainz.org/search_server/trunk search_server |
svn co http://svn.musicbrainz.org/search_server/trunk search_server |
||
cd mmd-schema/brainz-mmd2-jaxb |
cd mmd-schema/brainz-mmd2-jaxb |
||
Line 85: | Line 108: | ||
==Configure postgressql for better performance== |
==Configure postgressql for better performance== |
||
<pre> |
<pre> |
||
vi /etc/ |
sudo vi /etc/postgresql/9.1/main/postgresql.conf |
||
</pre> |
</pre> |
||
set shared_buffers=800MB |
- set shared_buffers=800MB |
||
set work_mem=20MB |
- set work_mem=20MB |
||
⚫ | |||
⚫ | |||
<pre> |
|||
sudo sysctl -w kernel.shmmax=17179869184 |
|||
sudo sysctl -w kernel.shmall=17179869184 |
|||
sudo service postgresql stop |
|||
sudo service postgresql start |
|||
</pre> |
|||
TODO:Add to startup so don't have to redo on every reboot |
|||
==Build Indexes== |
==Build Indexes== |
||
<pre> |
<pre> |
||
cd /home/paul/code/musicbrainz/search_server/index/target |
cd /home/paul/code/musicbrainz/search_server/index/target |
||
java -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --indexes-dir /home/paul/indexdata -d musicbrainz -u musicbrainz -p musicbrainz |
java -Xmx500M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --indexes-dir /home/paul/indexdata -d musicbrainz -u musicbrainz -p musicbrainz |
||
</pre> |
</pre> |
||
==Install Jetty server== |
|||
<pre> |
|||
sudo apt-get install jetty (note only install old jetty 6, but will do for now) |
|||
vi /etc/init.d/jetty (Remove JETTY_STOP line) |
|||
</pre> |
|||
==Deploy Search Server onto Jetty== |
|||
<pre> |
|||
cd /usr/share/jetty/webapps |
|||
rm -r ROOT |
|||
sudo cp /home/paul/code/musicbrainz/search_server/servlet/target/searchserver.war ROOT.war |
|||
sudo /etc/init.d/jetty start |
|||
</pre> |
|||
[[Category:How To]] |
Latest revision as of 14:50, 1 March 2022
Status: This Page is Glorious History!
This refers to very old MusicBrainz infrastructure including an older, barely related version of the search server. |
The existing instructions are okay, but because they try and cover all bases they can be more confusing than they need to be. So I thought it would be useful to show exactly what is required for just one common scenario :ijabz.
Install software not available from apt
- Download Desktop ISO image from http://xubuntu.org/ (I used Xubuntu 12.10 but I expect later versions should work) and burn to CD
- Install Xbuntu from the CD
- Download and unzip mbslave from https://bitbucket.org/lalinsky/mbslave
- Download an unzip psycopg from http://initd.org/psycopg/ (I used pyscogs 2.4.6 but later versions should work)
Download MusicBrainz Data Dumps
Download Musicbrainz data dumps from:
- US mirror: http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/
- Danish mirror: http://mirrors.dotsrc.org/MusicBrainz/data/fullexport/
cp /home/paul/Downloads/mbdump*.bz2 /home/paul/Downloads/lalinsky-mbslave*
Install Postgres
sudo apt-get install postgresql-9.1 postgresql-server-dev-9.1 postgresql-contrib-9.1 export PATH=$PATH:/usr/lib/postgresql/9.1/bin
Configue psycogs2
sudo apt-get install python python-psycopg2
Configure mbslave
cd cd Downloads/lalinsky-mbslave* cp mbslave.conf.default mbslave.conf
Uncomment password line and set to musicbrainz
Setup MusicBrainz Database
sudo su - postgres
and enter root password
createuser musicbrainz
Type y when asked if want to set up as superuser
createdb -l C -E UTF-8 -T template0 -O musicbrainz musicbrainz psql musicbrainz ALTER ROLE musicbrainz WITH PASSWORD 'musicbrainz'; Cntl-D
TODO Configure pg_conf file so that can connect to plsql using musicbrainz from any unix account
cd /home/paul/Downloads/lalinsky-mbslave* Put the following into a shell script, then run:
#!/bin/bash set -xv echo 'CREATE SCHEMA musicbrainz;' | ./mbslave-psql.py -S echo 'CREATE SCHEMA statistics;' | ./mbslave-psql.py -S echo 'CREATE SCHEMA cover_art_archive;' | ./mbslave-psql.py -S ./mbslave-remap-schema.py <sql/CreateTables.sql | sed 's/CUBE/TEXT/' | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/statistics/CreateTables.sql | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/caa/CreateTables.sql | ./mbslave-psql.py ./mbslave-import.py mbdump.tar.bz2 mbdump-derived.tar.bz2 mbdump-cdstubs.tar.bz2 mbdump-cover-art-archive.tar.bz2 ./mbslave-remap-schema.py <sql/CreatePrimaryKeys.sql | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/statistics/CreatePrimaryKeys.sql | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/caa/CreatePrimaryKeys.sql | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/CreateIndexes.sql | grep -vE '(collate|page_index|gist)' | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/statistics/CreateIndexes.sql | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/caa/CreateIndexes.sql | ./mbslave-psql.py ./mbslave-remap-schema.py <sql/CreateSimpleViews.sql | ./mbslave-psql.py
Exit out of postgres user
Cntl-D
Build Search Server
sudo apt-get install subversion openjdk-7-jdk maven2 tomcat7 git cd mkdir indexdata mkdir -p code/musicbrainz cd code/musicbrainz git clone https://github.com/metabrainz/mmd-schema.git mmd-schema svn co http://svn.musicbrainz.org/search_server/trunk search_server cd mmd-schema/brainz-mmd2-jaxb mvn install cd cd code/musicbrainz/search_server/servlet/src/main/webapp/WEB-INF
Edit web.xml change to <param-value>/home/paul/indexdata</param-value>
mvn package
Configure postgressql for better performance
sudo vi /etc/postgresql/9.1/main/postgresql.conf
- set shared_buffers=800MB
- set work_mem=20MB
- set effective_cache_size = 250MB
sudo sysctl -w kernel.shmmax=17179869184 sudo sysctl -w kernel.shmall=17179869184 sudo service postgresql stop sudo service postgresql start
TODO:Add to startup so don't have to redo on every reboot
Build Indexes
cd /home/paul/code/musicbrainz/search_server/index/target java -Xmx500M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --indexes-dir /home/paul/indexdata -d musicbrainz -u musicbrainz -p musicbrainz
Install Jetty server
sudo apt-get install jetty (note only install old jetty 6, but will do for now) vi /etc/init.d/jetty (Remove JETTY_STOP line)
Deploy Search Server onto Jetty
cd /usr/share/jetty/webapps rm -r ROOT sudo cp /home/paul/code/musicbrainz/search_server/servlet/target/searchserver.war ROOT.war sudo /etc/init.d/jetty start