History:Database Installation: Difference between revisions

From MusicBrainz Wiki
Jump to navigationJump to search
(migration fix (Imported from MoinMoin))
 
m (Database Installation moved to History:Database Installation: Out of date and probably confusing. Let's move it back if it's ever updated)
 
(22 intermediate revisions by 9 users not shown)
Line 1: Line 1:
<small>[[Products]] > [[Database]] > [[Database Setup|Database setup]] > Database setup on Linux </small>
This information on installing the [[MusicBrainz Database|MusicBrainzDatabase]] was [http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-June/001140.html provided] by [[User:DaveEvans|DaveEvans]] in response to [[Debian VMWare Database|DebianVMWareDatabase]].


'''Important: This document is outdated and may soon be deleted. Please follow [[Server Setup|these instructions for setting up a MusicBrainz server]] instead -- the instructions below are not up to date!'''
----


If you just want the data in a database, and you don't want Apache etc., here's another method.
* It is more flexible (you can choose not to bother creating the foreign key constraints, for example)

* It has fewer prerequisites (you do not need Perl, nor any of the Perl modules, nor postgresql-devel)
If you just want the data in a database, and you don't want Apache etc, here's another method.
* It's more flexible (you can choose not to bother creating the foreign key constraints, for example)
* It has fewer prerequisites (you don't need Perl, nor any of the Perl modules, nor postgresql-devel)
* However it is a bit more "hands on" - you have to check each step for errors yourself, and the import is about 10 or so commands instead of just one.
* However it is a bit more "hands on" - you have to check each step for errors yourself, and the import is about 10 or so commands instead of just one.


Here's the method:
==Here's the method:==

Install the base system. Install [[PostgreSQL]] (you need both the server and the client, if your system provides them separately). Install bzip2 and cvs.

Check mb_server out of CVS, and download mbdump*.tar.bz2, making sure they match. Configure postgresql the way you want it.

(Here mbrainz is the UNIX user I'm running as; musicbrainz_user is the PostgreSQL user I connect as; importtest is the database I'm setting up).

Create the empty database:

<pre>mbrainz$ createdb -U postgres --owner=musicbrainz_user
</pre>

<pre>- --encoding=UNICODE importtest
</pre>

<pre>CREATE DATABASE
</pre>

<pre>mbrainz$
</pre>

Create the empty tables:

<pre>mbrainz$ psql -U musicbrainz_user importtest
</pre>

<pre>Welcome to psql 7.4.8, the ["PostgreSQL"] interactive terminal.
</pre>

<pre>Type: \copyright for distribution terms
</pre>

<pre> \h for help with SQL commands
</pre>

<pre> \? for help on internal slash commands
</pre>

<pre> \g or terminate with semicolon to execute query
</pre>

<pre> \q to quit
</pre>

<pre>importtest=> \i admin/sql/CreateTables.sql
</pre>

<pre>BEGIN
</pre>

<pre>psql:admin/sql/CreateTables.sql:31: NOTICE: CREATE TABLE will create
</pre>

<pre>implicit sequence "album_id_seq" for "serial" column "album.id"
</pre>

<pre>CREATE TABLE
</pre>

... <pre>COMMIT
</pre>

<pre>importtest=> \q
</pre>

<pre>mbrainz$
</pre>

Decompress the mbdump*.tar.bz2 files:

<pre>mbrainz$ tar jxf /path/to/mbdump.tar.bz2
</pre>

(repeat for the other files too)

Import each file into the database:

<pre>mbrainz$ cd mbdump/
</pre>

<pre>mbrainz$ mkdir ../done
</pre>

<pre>mbrainz$ for t in * ; do echo `date` $t ; echo "\\copy $t from ./$t" |
</pre>

<pre>psql -U musicbrainz_user importtest && mv $t ../done/ ; done ; echo `date` Done
</pre>

<pre>Sun Jun 5 19:46:22 BST 2005 album
</pre>

... <pre>Sun Jun 5 19:51:06 BST 2005 url
</pre>

<pre>Sun Jun 5 19:51:09 BST 2005 Done
</pre>

<pre>mbrainz$ cd ..
</pre>

Add primary keys and indexes, and optimise the database:

<pre>mbrainz$ psql -U musicbrainz_user importtest
</pre>

<pre>Welcome to psql 7.4.8, the ["PostgreSQL"] interactive terminal.
</pre>

<pre>Type: \copyright for distribution terms
</pre>

<pre> \h for help with SQL commands
</pre>

<pre> \? for help on internal slash commands
</pre>

<pre> \g or terminate with semicolon to execute query
</pre>

<pre> \q to quit
</pre>


Install the base system. Install PostgreSQL (you need both the server and the client, if your system provides them separately). Install bzip2 and Subversion. [[Server Setup|Setup the MusicBrainz server]] (use the stable release) and download the [[Database Download|MusicBrainz Database]] dumps, please make sure their revision matches. Configure postgresql the way you want it.
<pre>importtest=> \i admin/sql/CreatePrimaryKeys.sql
: ''(Here mbrainz is the UNIX user I'm running as; <code><nowiki>musicbrainz_user</nowiki></code> is the PostgreSQL user I connect as; <code><nowiki>importtest</nowiki></code> is the database I'm setting up).''
</pre>


<ol><li> '''Create the empty database:'''
<pre>NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "album_pkey" for table "album"
<pre>mbrainz$ createdb -U postgres --owner=musicbrainz_user
</pre>
--encoding=UNICODE importtest
CREATE DATABASE
mbrainz$</pre>


<li> '''Create the empty tables:'''
<pre>ALTER TABLE
<pre>mbrainz$ psql -U musicbrainz_user importtest
</pre>
Welcome to psql 7.4.8, the ["PostgreSQL"] interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help on internal slash commands
\g or terminate with semicolon to execute query
\q to quit


importtest=> \i admin/sql/CreateTables.sql
... <pre>NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "wordlist_pkey" for table "wordlist"
BEGIN
</pre>
psql:admin/sql/CreateTables.sql:31: NOTICE: CREATE TABLE will create
implicit sequence "album_id_seq" for "serial" column "album.id"
CREATE TABLE...COMMIT
importtest=> \q
mbrainz$</pre></li>


<li> '''Decompress the mbdump*.tar.bz2 files:'''
<pre>ALTER TABLE
<pre>mbrainz$ tar jxf /path/to/mbdump.tar.bz2</pre>
</pre>
(repeat for the other files too) </li>


<li> '''Import each file into the database:'''
<pre>importtest=> \i admin/sql/CreateIndexes.sql
<pre>mbrainz$ cd mbdump/
</pre>
mbrainz$ mkdir ../done
mbrainz$ for t in * ; do echo `date` $t ; echo "\\copy $t from ./$t" |
psql -U musicbrainz_user importtest && mv $t ../done/ ; done ; echo `date` Done
Sun Jun 5 19:46:22 BST 2005 album
....
Sun Jun 5 19:51:06 BST 2005 url
Sun Jun 5 19:51:09 BST 2005 Done
mbrainz$ cd ..</pre></li>


<li> '''Add primary keys and indexes, and optimise the database:'''
<pre>CREATE INDEX
<pre>mbrainz$ psql -U musicbrainz_user importtest
</pre>
Welcome to psql 7.4.8, the ["PostgreSQL"] interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help on internal slash commands
\g or terminate with semicolon to execute query
\q to quit


importtest=> \i admin/sql/CreatePrimaryKeys.sql
<pre>CREATE INDEX
NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "album_pkey" for table "album"
</pre>
ALTER TABLE
...NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "wordlist_pkey" for table "wordlist"
ALTER TABLE


importtest=> \i admin/sql/CreateIndexes.sql
... <pre>CREATE INDEX
CREATE INDEX
</pre>
CREATE INDEX
...
CREATE INDEX
CREATE INDEX
importtest=> VACUUM ANALYZE;
importtest=> \q
mbrainz$</pre></li></ol>


That's it! If you want to, you could also choose to add the views ([http://bugs.musicbrainz.org/browser/mb_server/trunk/admin/sql/CreateViews.sql CreateViews.sql]). In fact since we're not running [http://bugs.musicbrainz.org/browser/mb_server/trunk/admin/sql/CreateFKConstraints.sql CreateFKConstraints.sql], the import is fairly quick too. If you wanted it even quicker, you might want to do without some of the indexes, and just add the ones you think you need.
<pre>CREATE INDEX
</pre>


If you're planning on writing to the database (instead of just querying from it), you might also want to set the correct sequence values (left as an exercise for the reader) and run [http://bugs.musicbrainz.org/browser/mb_server/trunk/admin/sql/CreateFKConstraints.sql CreateFKConstraints.sql], [http://bugs.musicbrainz.org/browser/mb_server/trunk/admin/sql/CreateFunctions.sql CreateFunctions.sql], and [http://bugs.musicbrainz.org/browser/mb_server/trunk/admin/sql/CreateTriggers.sql CreateTriggers.sql].
<pre>importtest=> VACUUM ANALYZE;
</pre>


~- This information on installing the [[Database|MusicBrainz Database]] was by [[User:DaveEvans|DaveEvans]] in response to [[Debian VMWare Database|DebianVMWareDatabase]] (see [http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-June/001140.html original post]).
<pre>importtest=> \q
</pre>


This method has been [http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-June/001149.html reported] to work on a Macintosh.
<pre>mbrainz$
</pre>


== Notes ==
That's it! If you want to, you could also choose to add the views ([[Create Views|CreateViews]].sql). In fact since we're not running [[Create FK Constraints|CreateFKConstraints]].sql, the import is fairly quick too. If you wanted it even quicker, you might want to do without some of the indexes, and just add the ones you think you need.


I had this annoying bug that my PosgreSql did not have the CUBE model. You can look at my blog post to see how I fixed it: http://omricohen.me/2011/05/03/musicbrainz-installation-of-postgresql-phppgadmin-and-importing-new-data/
If you're planning on writing to the database (instead of just querying from it), you might also want to set the correct sequence values (left as an exercise for the reader) and run [[Create FK Constraints|CreateFKConstraints]].sql, [[Create Functions|CreateFunctions]].sql, and [[Create Triggers|CreateTriggers]].sql.


----


<references />


This method has been [http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-June/001149.html reported] to work on a Macintosh.


[[Category:To Be Reviewed]]
[[Category:To Be Reviewed]] [[Category:Development]] [[Category:Database]]

Latest revision as of 08:50, 20 November 2011

Products > Database > Database setup > Database setup on Linux

Important: This document is outdated and may soon be deleted. Please follow these instructions for setting up a MusicBrainz server instead -- the instructions below are not up to date!

If you just want the data in a database, and you don't want Apache etc., here's another method.

  • It is more flexible (you can choose not to bother creating the foreign key constraints, for example)
  • It has fewer prerequisites (you do not need Perl, nor any of the Perl modules, nor postgresql-devel)
  • However it is a bit more "hands on" - you have to check each step for errors yourself, and the import is about 10 or so commands instead of just one.

Here's the method:

Install the base system. Install PostgreSQL (you need both the server and the client, if your system provides them separately). Install bzip2 and Subversion. Setup the MusicBrainz server (use the stable release) and download the MusicBrainz Database dumps, please make sure their revision matches. Configure postgresql the way you want it.

(Here mbrainz is the UNIX user I'm running as; musicbrainz_user is the PostgreSQL user I connect as; importtest is the database I'm setting up).
  1. Create the empty database:
    mbrainz$ createdb -U postgres --owner=musicbrainz_user
    --encoding=UNICODE importtest
    CREATE DATABASE
    mbrainz$
  2. Create the empty tables:
    mbrainz$ psql -U musicbrainz_user importtest
    Welcome to psql 7.4.8, the ["PostgreSQL"] interactive terminal.
    Type:  \copyright for distribution terms
           \h for help with SQL commands
           \? for help on internal slash commands
           \g or terminate with semicolon to execute query
           \q to quit
    
    importtest=> \i admin/sql/CreateTables.sql
    BEGIN
    psql:admin/sql/CreateTables.sql:31: NOTICE:  CREATE TABLE will create
    implicit sequence "album_id_seq" for "serial" column "album.id"
    CREATE TABLE...COMMIT
    importtest=> \q
    mbrainz$
  3. Decompress the mbdump*.tar.bz2 files:
    mbrainz$ tar jxf /path/to/mbdump.tar.bz2
    (repeat for the other files too)
  4. Import each file into the database:
    mbrainz$ cd mbdump/
    mbrainz$ mkdir ../done
    mbrainz$ for t in * ; do echo `date` $t ; echo "\\copy $t from ./$t" |
    psql -U musicbrainz_user importtest && mv $t ../done/ ; done ; echo `date` Done
    Sun Jun 5 19:46:22 BST 2005 album
    ....
    Sun Jun 5 19:51:06 BST 2005 url
    Sun Jun 5 19:51:09 BST 2005 Done
    mbrainz$ cd ..
  5. Add primary keys and indexes, and optimise the database:
    mbrainz$ psql -U musicbrainz_user importtest
    Welcome to psql 7.4.8, the ["PostgreSQL"] interactive terminal.
    Type:  \copyright for distribution terms
           \h for help with SQL commands
           \? for help on internal slash commands
           \g or terminate with semicolon to execute query
           \q to quit
    
    importtest=> \i admin/sql/CreatePrimaryKeys.sql
    NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "album_pkey" for table "album"
    ALTER TABLE
    ...NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "wordlist_pkey" for table "wordlist"
    ALTER TABLE
    
    importtest=> \i admin/sql/CreateIndexes.sql
    CREATE INDEX
    CREATE INDEX
    ...
    CREATE INDEX
    CREATE INDEX
    importtest=> VACUUM ANALYZE;
    importtest=> \q
    mbrainz$

That's it! If you want to, you could also choose to add the views (CreateViews.sql). In fact since we're not running CreateFKConstraints.sql, the import is fairly quick too. If you wanted it even quicker, you might want to do without some of the indexes, and just add the ones you think you need.

If you're planning on writing to the database (instead of just querying from it), you might also want to set the correct sequence values (left as an exercise for the reader) and run CreateFKConstraints.sql, CreateFunctions.sql, and CreateTriggers.sql.

~- This information on installing the MusicBrainz Database was by DaveEvans in response to DebianVMWareDatabase (see original post).

This method has been reported to work on a Macintosh.

Notes

I had this annoying bug that my PosgreSql did not have the CUBE model. You can look at my blog post to see how I fixed it: http://omricohen.me/2011/05/03/musicbrainz-installation-of-postgresql-phppgadmin-and-importing-new-data/