Difference between revisions of "History:Development/Search server setup"

From MusicBrainz Wiki
(Server setup for developers)
Line 1: Line 1:
 
=Server setup for developers=
 
=Server setup for developers=
 +
 +
The following instructions are aimed at getting a search server up and running as a developer; I will assume you are installing things into your home directory, and running under your current user account.  If you're looking for a more production ready setup follow the instructions in the README included with the search_server.
 +
 +
Some of the instructions below are taken from the search_server README.
  
 
==Requirements==
 
==Requirements==
Line 17: Line 21:
 
You will also need a freedb dump, which you can download from http://www.freedb.org/en/download__database.10.html .
 
You will also need a freedb dump, which you can download from http://www.freedb.org/en/download__database.10.html .
  
You will also need these tools:
+
And you will need these tools:
  
 
* Java complete with JDK 1.6, not just the JRE
 
* Java complete with JDK 1.6, not just the JRE
Line 36: Line 40:
 
     export JAVA_HOME="/usr/bin/java"
 
     export JAVA_HOME="/usr/bin/java"
  
To build you need to install Maven, if you havent already got it. Download
+
To build you need to install Maven, if you haven't already got it. Download it from http://maven.apache.org/download.html .
it from http://maven.apache.org/download.html .
 
  
 
==Download the search server and schemas==
 
==Download the search server and schemas==
Line 82: Line 85:
 
==Install the servlet==
 
==Install the servlet==
  
 +
To install the servlet in Apache Tomcat, you obviously need a tomcat install.  I downloaded a binary distribution and extracted in ~/opt.  If you have java in your path you should be able to start it right away.
 +
 +
  warp@bo-bomb:~/opt$ tar xfz apache-tomcat-6.0.24.tar.gz
 +
  warp@bo-bomb:~/opt$ cd apache-tomcat-6.0.24
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/startup.sh
 +
  Using CATALINA_BASE:  /home/warp/opt/apache-tomcat-6.0.24
 +
  Using CATALINA_HOME:  /home/warp/opt/apache-tomcat-6.0.24
 +
  Using CATALINA_TMPDIR: /home/warp/opt/apache-tomcat-6.0.24/temp
 +
  Using JRE_HOME:        /usr
 +
  Using CLASSPATH:      /home/warp/opt/apache-tomcat-6.0.24/bin/bootstrap.jar
 +
  warp@bo-bomb:~/opt$
 +
 +
Try connecting to http://localhost:8080, and you should see the tomcat startup page. 
 +
 +
Shut down tomcat again by running "bin/shutdown.sh".  To install the search server, enter the "webapps" directory, remove the current ROOT, and copy over the servlet as the new ROOT webapp:
 +
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ cd webapps/
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ ls
 +
  docs  examples host-manager  manager  ROOT
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ rm -rf ROOT
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ cp ~/code/search_server/servlet/target/searchserver.war ROOT.war
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ cd ..
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/startup.sh
 +
 +
You should see a new ROOT directory tree appear under webapps, stop tomcat again to configure the indexes for the search server.
 +
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/shutdown.sh
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ $EDITOR webapps/ROOT/WEB-INF/web.xml
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/startup.sh
 +
  warp@bo-bomb:~/opt/apache-tomcat-6.0.24$
 +
 +
In web.xml, you should replace the current index_dir value ("/home/search/indexdata") with the path to the index files, in my case this is "/home/warp/code/search_server/index/target/data".
  
10:21 < ijabz> java -Xmx 500m -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar -?
+
If you now query the search server, you should get some resultsFor example, try this: [[http://localhost:8080/ws/2/artist?query="alice"]].
10:21 < ijabz> shows the options
 
...
 
10:31 < ijabz> try java -Xmx512M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar -d musicbrainz_db -u warp -p (password) -s warp -w (password) -t
 
10:31 < warp> I just added the raw database fields, and it started doing stuff now.
 
  10:32 < ijabz> :), so have you got tomcat installed yet ?
 
10:32 < warp> yes
 
10:33 < warp> or well... I unpacked it and ran bin/startup.sh, and now something is listening on http://localhost:8080
 
10:33 < ijabz> Which tomcat download did you use ?
 
10:33 < warp> apache-tomcat-6.0.24
 
10:34 < ijabz> ok so you need to coy searchserver,.war into the webpps folder and rename to ROOT.war
 
10:35 < warp> I can do that while tomcat is running, right?
 
10:35 < ijabz> Yes
 
10:35 < ijabz> But do you have a ROOT folder within webaps
 
10:35 < warp> yes
 
10:36 < ijabz> ok, I think you have to shutdown and restart for change to take effect
 
10:36 < ijabz> bin/shutdown.sh; bin/startup.sh
 
...
 
10:44 < ijabz> And I meant to say you needed to delete that ROOT folder
 
10:44 < ijabz> so please do that, then shutdown and restart again
 
10:44 < warp> aah :)
 
10:45 < warp> yay, it created a new ROOT now.
 
10:48 < warp> HTTP Status 500 - Error during servlet initialization directory '/home/search/indexdata/artist_index' does not exist
 
10:49 < warp> ijabz: how do I tell it where my indexes are?
 
10:50 < ijabz> I thought you created them in the default location .... yo need to edit /webapps/ROOT/WEB-INF/web.xml
 
10:50 < warp> (where are my indexes? :)
 
...
 
10:52 < ijabz> I think index defaults to building them in a folder called data
 
10:52 < ijabz> from whereever you run it
 
10:52 < warp> ah, yes, there they are.
 
10:52 < ijabz> Ok, specify that path in web.xml then
 
10:52 < warp> do I have to shutdown tomcat for that?
 
10:53  * warp does so, just to be sure.
 
10:54 < alastairp> to change web.xml - yes
 
...
 
http://www.freedb.org/en/download__database.10.html
 

Revision as of 15:32, 23 February 2010

Server setup for developers

The following instructions are aimed at getting a search server up and running as a developer; I will assume you are installing things into your home directory, and running under your current user account. If you're looking for a more production ready setup follow the instructions in the README included with the search_server.

Some of the instructions below are taken from the search_server README.

Requirements

This server can be run on linux/osx/windows, but these instructions assume that you will be running linux and will be using the tomcat6 server to host the servlet.

To run this search server you will also need to have a MusicBrainz database with the core data set loaded. Please see

 FIXME: link to ngs server install here.

for details on how to setup at least a "database only install". This document assumes that you have completed the setup of this server.

You will also need a freedb dump, which you can download from http://www.freedb.org/en/download__database.10.html .

And you will need these tools:

In a recent Ubuntu, you can install these with this command:

   apt-get install openjdk-6-jdk maven2 tomcat6

Ensure default Java installation is at least 1.6, and that you have the JDK installed not just the JRE.

   java -version

If you're not on Ubuntu, you can get an official version from http://java.sun.com/javase/downloads and setup Java Home directory in your profile (not needed on ubuntu):

   export JAVA_HOME="/usr/bin/java"

To build you need to install Maven, if you haven't already got it. Download it from http://maven.apache.org/download.html .

Download the search server and schemas

You will obviously need the search server, but you also need the musicbrainz schema files:

svn co http://svn.musicbrainz.org/mmd-schema/trunk mmd-schema
svn co http://svn.musicbrainz.org/search_server/trunk search_server

Install the schema classes

Enter the brainz-mmd2-jaxb directory in mmd-schema, and run "mvn install".

 warp@bo-bomb:~/code/mmd-schema/brainz-mmd2-jaxb$ mvn install

You should see it install mmd2-2.0.0.jar in a directory where maven2 will be able to find it later:

 [INFO] Installing /home/warp/code/mmd-schema/brainz-mmd2-jaxb/target/mmd2-2.0.0.jar to /home/warp/.m2/repository/org/musicbrainz/mmd2/mmd2/2.0.0/mmd2-2.0.0.jar

Build the search server servlet

Enter the search_server directory and run "mvn package" to build the servlet.

You should see a line like this when it's done:

[INFO] Building war: /home/warp/code/search_server/servlet/target/searchserver.war

Build the indexes

To build the indexes, the search server will need access to both your MusicBrainz database and the freedb dump. First enter the index/target directory and run "java -Xmx512M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar -?" to see the options.

Figure out your database name and login details (if you already have mb_server running you can find them in lib/DBDefs.pm in the mb_server directory).

 warp@bo-bomb:~/code/search_server/index/target$ java -Xmx512M -jar index-2.0-SNAPSHOT-jar-with-dependencies.jar --db-name musicbrainz_db --db-user warp --db-password mypassword --raw-db-name musicbrainz_db_raw --raw-db-user warp --raw-db-password mypassword --freedb-dump ~/Downloads/freedb-complete-20100201.tar.bz2 --test

Adding '--test' at the end will build a small test index. After you are done with this setup, you may want to re-run this command without the '--test' option to build the full indexes.

When it is done you should have the following index directories in the data/ dir:

 warp@bo-bomb:~/code/search_server/index/target$ ls data/
 annotation_index  cdstub_index	label_index	 releasegroup_index  work_index
 artist_index	  freedb_index	recording_index  release_index

Install the servlet

To install the servlet in Apache Tomcat, you obviously need a tomcat install. I downloaded a binary distribution and extracted in ~/opt. If you have java in your path you should be able to start it right away.

 warp@bo-bomb:~/opt$ tar xfz apache-tomcat-6.0.24.tar.gz 
 warp@bo-bomb:~/opt$ cd apache-tomcat-6.0.24
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/startup.sh 
 Using CATALINA_BASE:   /home/warp/opt/apache-tomcat-6.0.24
 Using CATALINA_HOME:   /home/warp/opt/apache-tomcat-6.0.24
 Using CATALINA_TMPDIR: /home/warp/opt/apache-tomcat-6.0.24/temp
 Using JRE_HOME:        /usr
 Using CLASSPATH:       /home/warp/opt/apache-tomcat-6.0.24/bin/bootstrap.jar
 warp@bo-bomb:~/opt$ 

Try connecting to http://localhost:8080, and you should see the tomcat startup page.

Shut down tomcat again by running "bin/shutdown.sh". To install the search server, enter the "webapps" directory, remove the current ROOT, and copy over the servlet as the new ROOT webapp:

 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ cd webapps/
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ ls
 docs  examples	host-manager  manager  ROOT
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ rm -rf ROOT
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ cp ~/code/search_server/servlet/target/searchserver.war ROOT.war
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24/webapps$ cd ..
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/startup.sh 

You should see a new ROOT directory tree appear under webapps, stop tomcat again to configure the indexes for the search server.

 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/shutdown.sh 
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ $EDITOR webapps/ROOT/WEB-INF/web.xml 
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ bin/startup.sh 
 warp@bo-bomb:~/opt/apache-tomcat-6.0.24$ 

In web.xml, you should replace the current index_dir value ("/home/search/indexdata") with the path to the index files, in my case this is "/home/warp/code/search_server/index/target/data".

If you now query the search server, you should get some results. For example, try this: ["alice"].