MusicBrainz Server/Setup: Difference between revisions

From MusicBrainz Wiki
Jump to navigationJump to search
(Add hints on how to run the VM with qemu; I couldn't get it to boot with the kvm hypervisor.)
m (→‎Accessing the database: Add default password for the database from INSTALL.md)
(46 intermediate revisions by 12 users not shown)
Line 1: Line 1:
== MusicBrainz Server virtual machine ==
== MusicBrainz Server virtual machine ==


Running a MusicBrainz Server as a virtual machine requires some Linux knowledge, but it is vastly simpler than installing the server from scratch. The pre-built virtual image can be imported into either VirtualBox or VMware. If you are using Amazon EC2 you can not use this virtual image and will instead have to follow the steps outlined in the source code [https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md INSTALL.md]. To set up a virtual machine instance, download the torrent file and follow these steps:
Running a MusicBrainz Server as a virtual machine requires some Linux knowledge, but it is vastly simpler than installing the server from scratch. (Have a look at the [http://tickets.musicbrainz.org/browse/MBVM open issues in the bug tracker], though.) The pre-built virtual image can be imported into either VirtualBox or VMware. If you are using Amazon EC2 you can not use this virtual image and will instead have to follow the steps outlined in the source code [https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md INSTALL.md]. To set up a virtual machine instance, download the torrent file and follow these steps:


<div style="background-color:#FFFAF5; border: 1px solid #FFDAA4; padding:10px 20px; width: 400px; margin: 20px 20px 20px 0">
<div style="background-color:#FFFAF5; border: 1px solid #FFDAA4; padding:10px 20px; width: 448px; margin: 20px 20px 20px 0">
'''MusicBrainz Server Virtual Machine <br />
'''MusicBrainz Slave Server Virtual Machine (Virtual Box)<br />'''
Version: 2018-08-14<br />
Download via BitTorrent: [http://users.musicbrainz.org/~robert/MusicBrainz%20VM%202013-08-01.torrent MusicBrainz VM 2013-08-01]<br />
Size: 10.13GB Open Virtualization Archive (OVA)<br/>
Size: 22.16GB<br />
Format: Open Virtualization Archive (OVA)<br />
(the virtual drive can extend up to 40 GB)<br/>
Download via either [ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-08-14.ova.torrent BitTorrent] or [ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/musicbrainz-server-2018-08-14.ova FTP]<br />
Version: 2013-08-01 (beta version)<br />
MD5: <small>717726cbea28a4fce51ae6ae803b8a6a</small>
MD5: <small>e566ce903e14a7db6251a346799b5bf9</small>
<hr />
Data:
* Database schema sequence number 24
* Full export 20180811-001618
Software:
* MusicBrainz server v-2018-08-14
* Lucene-based MusicBrainz search server/indexer f297b72/a63d655
* Redis 3.2.12
* PostgreSQL 9.5.14
Based on the bento box v201807.12.0 of Ubuntu 18.04 LTS (Bionic)
</div>
</div>


=== Running with VMware ===
=== Running with VMware ===

# Download [http://www.vmware.com/products/player/overview.html VMware Player] for Windows/Linux or [http://www.vmware.com/products/fusion/overview.html VMware Fusion] for Mac.
We no longer support running our VM under VMWare, since VirtualBox has become stable enough for our needs. If you intend to run this with other VM software, we doubt it will work and we're not going to support other software.
# Import the downloaded .ova


=== Running with VirtualBox ===
=== Running with VirtualBox ===
Currently (20130801 beta VM) the .ova file doesn't work directly with VirtualBox so you have to do some additional steps until a fixed VM is realeased.
# Download and install [http://virtualbox.org VirtualBox] on your machine.
# Extract the virtual disk (vmdk) from the ova file and create a VirtualBox hard disk (vdi). This can be done with:<pre>tar -xf "MusicBrainz 2013-08-01.ova" && VBoxManage clonehd MusicBrainz_2013-08-01-disk1.vmdk MusicBrainz130801.vdi --format VDI</pre>This command will create a 27 GB file (the virtual hard drive), which can grow up to 40 GB later on when using the VM.<br/>Make sure you don't use the vmdk directly after extracting it, since the VBoxManage command might fail otherwise. If that happens you should untar the .ova again to get an untouched vmdk.
# Create a new VirtualBox VM with Type "Linux" and Version "Ubuntu (64 bit)". You should give at least 1 GB RAM.
# Add the previously generated MusicBrainz130801.vdi as hard disk
<!-- (usual commands commented out)
# Start VirtualBox and choose ''Import Appliance'' from the File menu. Select the downloaded file.</strike>
# <strike>Once VirtualBox has imported the appliance, select the imported virtual machine from the list of virtual machines and click on Start.
//-->


Before you start, make sure you have the lastest version of [https://www.virtualbox.org/wiki/Downloads VirtualBox] installed.
=== Running with QEMU ===

# Extract the disk image: <pre>tar -xf MusicBrainz\ 2013-08-01.ova</pre>
# Start VirtualBox and choose ''Import Appliance'' from the File menu. Select the downloaded file.
# Covert it from VMWare format: <pre>virt-convert -D qcow2 MusicBrainz\ 2013-08-01.ovf /var/lib/libvirt/images</pre>
# Once VirtualBox has imported the appliance, select the imported virtual machine from the list of virtual machines and click on Start.
# Create and start the new VM: <pre>virt-install --virt-type qemu --os-type=linux --os-variant=ubuntuprecise --arch x86_64 --ram 1024 --name MusicBrainz_2013-08-01 --import --disk /var/lib/libvirt/images/MusicBrainz_2013-08-01-disk1.qcow2</pre>

{{Note|Troubleshooting version 2018-08-14, which is based on Ubuntu 18.04 LTS (Bionic) with Linux 4.15.0-20-generic, if the directory <code>/home/vagrant/bin</code> is missing, fix your VirtualBox setup or upgrade your Ubuntu system from 16.04 (Xenial) to 18.04 (Bionic).}}

=== Using the VM ===

# Once the instance has started up, log in on the console using the username ''vagrant'' and password ''vagrant''. This account has sudo privileges -- if you would like to set a root password, you can do that via sudo.

# You can SSH into the machine via the port 2222 on localhost of the host OS. (e.g. ssh -p 2222 vagrant@localhost )
# To access the MusicBrainz web site and web service, go to: http://localhost:5000
# To access the Search Server web service, go to: http://localhost:8080


=== Starting the VM ===
=== Resetting Docker Containers ===
This new VM is based on docker containers. The docker containers ''should'' start automagically at boot time, but sometimes they don't. If any of the commands listed below do not work or give you strange docker/container errors, reset the containers with this command from the VM:
# Once the instance has started up, log in on the console using the username ''vm'' and password ''musicbrainz''. This account has sudo privileges -- if you would like to set a root password, you can do that via sudo.
bin/reset-containers
# Get the IP address of your virtual machine. Note this down from "inet addr" with this command: <pre>ifconfig | grep eth0 -A 1</pre> NOTE: if eth0 is not configured correctly, perhaps you've encountered [http://superuser.com/questions/90584/network-interface-missing-in-ubuntu-after-changing-mac-address this problem].
Then try your command again.
# Optional: The console for Virtual Box is very slow. It may be faster to SSH into the virtual box with a good terminal program.
# The MusicBrainz server starts automatically. Now you can reach the MusicBrainz server by pointing your browser to port 5000 of the IP address in step 5.
#* e.g If your IP address from step 6 was: 10.1.1.104, then point your browser to http://10.1.1.104:5000
#* If the server looks like it started correctly and you cannot connect in your browser, try pinging it from your '''host''' operating system command line <pre>ping <inet addr></pre> If you cannot ping it, you may have a VirtualBox network configuration problem. Try the instructions [http://askubuntu.com/questions/56434/cant-ping-from-virtualbox-ubuntu-server-nat-connection here] to switch your VM to using a ''bridged connection'' rather than ''NAT''. Restart the virtual machine and find the new IP address in step 6 again.


=== Tuning your VM ===
=== Tuning your VM ===


We recommend that you give your VM 2GB of ram, if that is possible. The more RAM you give to the VM, the faster it is going to run. To change the memory settings, you will need to shut down the VM, change the settings and then re-start the VM. The Postgres database will be automatically tuned based on the available RAM when the VM starts up.
We recommend that you give your VM 4GB of ram and 2 CPU cores, if that is possible. The more RAM you give to the VM, the faster it is going to run. To change the memory settings, you will need to shut down the VM, change the settings and then re-start the VM.


=== Running Replication ===
=== Running Replication ===


This VM comes "replication ready". To enable replication, and have the database catch up with the latest replication packets, do this:
This VM comes "replication ready". To enable replication, and have the database catch up with the latest replication packets, you will need to sign up for a Live Data Feed access token on the [https://metabrainz.org/supporters/account-type MetaBrainz web site]. Then SSH into the machine as described above then:

bin/set-token <replication token>

To start replication, do this:

bin/replicate now
bin/replicate now


This will load all of the changes to the database since the VM update.
This will load all of the changes to the database since the VM update. Unlike the previous versions of this VM, this command will not give you any output as it applies replication packets. If you want to see the progress, open another terminal window to the VM and then:

bin/tail-replication-log


This will allow you to see the output of the replication packets being applied.
NOTE: Loading replication changes might take a long time. If the VM is more than a couple of weeks old, it might be better for you to import a [[MusicBrainz Database/Download|fresh data set]]. Drop pre-filled database using command <pre>dropdb musicbrainz_db</pre> and check [https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md the INSTALL.md file] for how to import new data.


=== Automating Replication ===
=== Automating Replication ===
Line 69: Line 84:
The VM comes with support to build search indexes. In order to build the indexes, log in to the account and then:
The VM comes with support to build search indexes. In order to build the indexes, log in to the account and then:


bin/reindex
sudo bin/reindex


Depending on your machine, this may take quite a long time. We recommend that you leave this running overnight. After the indexes are complete, you should be able to carry out indexed searches in your VM.
Depending on your machine, this may take quite a long time. We recommend that you leave this running overnight. After the indexes are complete, you should be able to carry out indexed searches in your VM.

{{Note|Troubleshooting version 2018-08-14, running `sudo` as above is a workaround for a bug in `bin/reindex` that can be fixed by running once:}}

<code>sudo sed -i '/sudo/!s/ls\|rm/sudo &/' bin/reindex</code>


=== Accessing the database ===
=== Accessing the database ===


To access the main postgres database, you can do this:
To access the main postgres database from the outside, first run once (from the VM):


bin/turn-port db on
sudo su - musicbrainz
cd musicbrainz-server/admin
./psql READWRITE


{{Note|Troubleshooting version 2018-08-14: Recreate the <code>db</code> container after running <code>turn-port</code> by running (from the VM):}}
If you would like to access the DB from outside the virtual box, take a look at [http://www.cyberciti.biz/tips/postgres-allow-remote-access-tcp-connection.html how to change postgres connection settings].

docker-compose -f ~/musicbrainz/musicbrainz-docker/docker-compose.yml up -d db

then you can do this (from the host):

psql -h localhost -U musicbrainz -p 15432

You'll need to install the ''psql'' postgres command line client on your host OS.

By default, the password for the user musicbrainz is <code>musicbrainz</code>, as specified in the doc “[https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md#creating-the-database Installing MusicBrainz Server]”.

=== Accessing from a remote host ===

To access the MusicBrainz web site and web service from a remote host, run once (from the VM):

bin/set-web-server-name ''something.that.works''

where ''something.that.works'' can be an IP address, or a host name, or a fully qualified domain name.

Then go to: <nowiki>http://</nowiki>''something.that.works'':5000

{{Note|Troubleshooting version 2018-08-14, the following warning message can safely be ignored
(node:80) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 12 resolve listeners added. Use emitter.setMaxListeners() to increase limit
}}


=== Turning the VM into development box ===
=== Turning the VM into development box ===

If you need neither replication nor search, you may prefer to use the lighter virtual machine that contains sample data only.

<div style="background-color:#FFFAF5; border: 1px solid #FFDAA4; padding:10px 20px; width: 400px; margin: 20px 20px 20px 0">
'''MusicBrainz Sample Virtual Machine (Virtual Box)<br />'''
Version: 2018-08-14<br />
Size: 3.26GB<br />
Format: Open Virtualization Archive (OVA)<br />
Download via either [ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/sample/musicbrainz-sample-2018-08-14.ova.torrent BitTorrent] or [ftp://ftp.eu.metabrainz.org/pub/musicbrainz-vm/sample/musicbrainz-sample-2018-08-14.ova FTP]<br />
MD5: <small>5bd8714b59c32a7199e4d33446cceb78</small>
</div>


If you would like to use the VM to do development instead of using it as a simple database slave, you'll need to edit lib/DBDefs.pm and set REPLICATION_TYPE to RT_STANDALONE and run admin/psql READWRITE and execute the following queries:
If you would like to use the VM to do development instead of using it as a simple database slave, you'll need to edit lib/DBDefs.pm and set REPLICATION_TYPE to RT_STANDALONE and run admin/psql READWRITE and execute the following queries:
Line 95: Line 147:
admin/psql READWRITE < admin/sql/CreateFunctions.sql
admin/psql READWRITE < admin/sql/CreateFunctions.sql


== Migrating the MusicBrainz Server from a Virtual Machine to a Physical Machine ==
TODO: The server will probably run out of disk space during this process. We need to add instructions on how to move the DB to a new partition.
Taken from [http://askubuntu.com/questions/32499/migrate-from-a-virtual-machine-vm-to-a-physical-system AskUbuntu].

It is possible to migrate the MusicBrainz Server image (or any virtual image) from a virtual machine onto bare metal, should you desire to.

=== Requirements ===
* A LiveCD or LiveUSB (this walkthrough will assume you are using a Ubuntu variant as that is what the image is based on.) '''If you are using an image from 2013-08-01 or earlier, you must have a Ubuntu 12.04.1 Live media.'''
* USB media large enough to store the raw disk image for writing on a new computer.
* Patience

=== Step 1: Converting the Disk to a Raw format ===
This step is included mostly for the sake of convenience. By doing some prep work on the image before writing, standard Linux tools can be used to write to and manipulate the disk afterwards. This can be done from your current host environment without booting to your LiveCD.
* Load a terminal
cd /path/to/image/
# for VMWare
sudo apt-get install qemu-kvm
qemu-img convert your-vmware-disk.vmdk -O raw disk.img
# For VirtualBox
VBoxManage internalcommands converttoraw your-virtualbox-disk.vdi disk.img

* Move <code>disk.img</code> to somewhere that you're not about to write to. If your target is another physical machine, then an external medium is probably the most convenient way to move the image. These instructions assume that you have moved it to <code>/media/external/disk.img</code>. Adjust accordingly.
* Before you do anything permanent, '''make sure you do backups'''. These instructions will erase anything on the target disk or partition. It's better to be safe than sorry.

=== Option 1: Write the Image to it's Own Disk ===
'''This assumes that you're going to overwrite an entire disk'''. This is the most ideal situation since the server will have an entire disk to work with. If you're looking to have the image installed along side an existing operating system, skip to Option 2.
* Boot into your LiveCD and Click Try Ubuntu.
* Mount the volume where <code>disk.img</code> resides to somewhere safe (like /media/external).
* Determine which volume is your target using a command like <code>lsblk</code> or <code>blkid</code>
* Then we copy the contents of the image to the drive Replace <code>sdX</code> with the correct destination disk. The password for <code>sudo</code> is blank.
sudo dd if=/media/external/disk.img of=/dev/sdX
* After this has finished, you can use <code>gparted</code> or another utility to manipulate the partition table to the desired layout. It is recommended to have a sizeable swap partition (see [http://www.cyberciti.biz/tips/linux-swap-space.html this article] for some recommendations on how big it should be) as well as expand the root partition to use all remaining available space on the disk. If you desire a seperate boot partition for the system as well, set aside some space here. (Steps for this can be found [https://help.ubuntu.com/community/BootPartition in the Ubuntu community documentation])
* '''If you are using an image from 2013-08-01 or earlier, please see [[ChangingKernelVersions|here]]'''.

=== Option 2: Installing the Image Alongside Another OS ===
This is potentially safer than the above method and is very similar. Your LiveCD '''must be the same Ubuntu version as the virtual image.'''
* Boot into the LiveCD and click Install Ubuntu.
* Follow the installer prompts, repartition things as you see fit. When the installer finishes, it will ask you to reboot. Do not reboot. You must be in the Live environment for the next instructions.
* Mount your new Ubuntu install (target) and your external disk where you stored <code>disk.img</code> earlier.
* Mount <code>disk.img</code>.
sudo mkdir /media/oldinstall
sudo mount -o loop /media/external/disk.img /media/oldinstall
* From there you can either cherry pick files or copy over everything on top of your target. For example, using <code>rsync</code>:
sudo rsync /media/oldinstall /media/newinstall
* Reboot and you ''should'' be greeted by your virtual machine, but on bare metal. If you have GRUB errors, see [https://help.ubuntu.com/community/Grub2#Reinstalling%20GRUB%202 this article from the Ubuntu community documentation]

=== <div id="ChangingKernelVersions">Changing Kernel Versions</div> ===
Parts taken from [https://help.ubuntu.com/community/Grub2/Installing#via_ChRoot the Ubuntu community documentation].

For images from 2013-08-01 and older, you will need to change the kernel flavour from -virtual to -generic. The -virtual kernel flavour is lighter, but only supports hardware typically seen in virtual environments while -generic provides support for a wider range of hardware, such as what you would see in a physical server. Both of these versions are provided in the default Ubuntu repositories.
* Boot into your LiveCD. Click "Try Ubuntu"
* Open a terminal.
* Determine which disk is yours using a command like <code>sudo lsblk</code> or <code>sudo blkid</code>.
* Mount your root partition. '''X''' is the drive letter and '''Y''' is the partition number:
sudo mount /dev/sdXY /mnt
** If you have a seperate boot partition, mount it as well ('''Z''' is the boot partition number):
sudo mount /dev/sdXZ /mnt/boot
* Mount the virtual filesystems using the following single command:
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
* Chroot into your installation
sudo chroot /mnt
* Install the latest -generic image and headers by installing the metapackage. This will regenerate GRUB's configuration.
sudo apt-get install linux-image-generic linux-header-generic
* Purge the -virtual image and headers. This should also regenerate GRUB's configuration:
sudo apt-get purge --auto-remove linux-image-virtual linux-header-virtual
* Check for leftover -virtual images in <code>/boot</code>. If you find any, remove them and then run <code>sudo update-grub</code>
* Reboot into your installation.


== Setup MusicBrainz Server from source code ==
== Setup MusicBrainz Server from source code ==
This can potentially be a very laborious and time consuming method of getting a functioning MusicBrainz server. Using the virtual machine is recommended.
This can potentially be a very laborious and time consuming method of getting a functioning MusicBrainz server. Using the virtual machine is recommended, except for development purpose.


Get a copy of musicbrainz-server from git:
Get a copy of musicbrainz-server from git:
Line 105: Line 222:
And follow the instructions in the [https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md INSTALL] file.
And follow the instructions in the [https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md INSTALL] file.


== Support ==
=== Support ===


The setup process may look daunting, but please don't let this discourage you; the INSTALL is thorough and contains a lot of information, and we are willing to provide assistance. If you have questions about installing, join us in the [http://webchat.freenode.net/?channels=musicbrainz-devel #musicbrainz-devel IRC channel] or post a question on the [[Developers Mailing List|developers mailing list]] and we will attempt to help you out.
The setup process may look daunting, but please don't let this discourage you; the INSTALL is thorough and contains a lot of information, and we are willing to provide assistance. If you have questions about installing, join us in the [http://webchat.freenode.net/?channels=metabrainz #metabrainz IRC channel] or post a question on the [https://community.metabrainz.org/tags/musicbrainz-server community forum] and we will attempt to help you out.


We recommend that you dive in and give it a try - who knows how far you'll get and what you might learn along the way!
We recommend that you dive in and give it a try - who knows how far you'll get and what you might learn along the way!


==Requirements==
===Requirements===


In order to set up a running MusicBrainz server with the full database you will need:
In order to set up a running MusicBrainz server with the full database you will need:
* A linux box, preferably Ubuntu.
* A linux box, preferably Ubuntu.
* 8GB of free disk space, (if you are a developer and only want the server code and database structure 2GB is more than enough).
* 20GB+ of free disk space, (if you are a developer and only want the server code and database structure somewhat less should do).
* [[Git]] knowledge which will enable you to check out the source code.
* [[Git]] knowledge which will enable you to check out the source code.
* PostgreSQL 9.5 or later (and some [https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md#prerequisites other dependencies])


As a developer the following knowledge/skills are beneficial:
As a developer the following knowledge/skills are beneficial:
* Apache, Perl, mod_perl, PostgreSQL and a number of perl modules.
* Perl and a number of perl modules, PostgreSQL, nginx.
* How to compile and install packages from source on a Linux box.
* How to compile and install packages from source on a Linux box.
* How to patch existing packages, although we can help you out if you have questions about that.
* How to patch existing packages, although we can help you out if you have questions about that.


Note: The server has ''never been ported to Windows'', and we suspect that it would be a fair amount of work to make that happen.
Note: The server has ''never been ported to Windows'', and we suspect that it would be a fair amount of work to make that happen.

==External Links==
* [https://github.com/metabrainz/musicbrainz-server MusicBrainz Server] on GitHub
* List of [http://blog.musicbrainz.org/category/server/?tag=changelog server updates] on the MusicBrainz blog
* List of supplemental [http://blog.musicbrainz.org/category/server/?tag=instructions instructions] on the MusicBrainz blog
* List of [http://blog.musicbrainz.org/category/virtual-machine/ VM releases] on the MusicBrainz blog


[[Category:WikiDocs Page]] [[Category:Download]] [[Category:Development]] [[Category:Server]]
[[Category:WikiDocs Page]] [[Category:Download]] [[Category:Development]] [[Category:Server]]

Revision as of 16:38, 25 October 2018

MusicBrainz Server virtual machine

Running a MusicBrainz Server as a virtual machine requires some Linux knowledge, but it is vastly simpler than installing the server from scratch. (Have a look at the open issues in the bug tracker, though.) The pre-built virtual image can be imported into either VirtualBox or VMware. If you are using Amazon EC2 you can not use this virtual image and will instead have to follow the steps outlined in the source code INSTALL.md. To set up a virtual machine instance, download the torrent file and follow these steps:

MusicBrainz Slave Server Virtual Machine (Virtual Box)
Version: 2018-08-14
Size: 22.16GB
Format: Open Virtualization Archive (OVA)
Download via either BitTorrent or FTP
MD5: e566ce903e14a7db6251a346799b5bf9


Data:

  • Database schema sequence number 24
  • Full export 20180811-001618

Software:

  • MusicBrainz server v-2018-08-14
  • Lucene-based MusicBrainz search server/indexer f297b72/a63d655
  • Redis 3.2.12
  • PostgreSQL 9.5.14

Based on the bento box v201807.12.0 of Ubuntu 18.04 LTS (Bionic)

Running with VMware

We no longer support running our VM under VMWare, since VirtualBox has become stable enough for our needs. If you intend to run this with other VM software, we doubt it will work and we're not going to support other software.

Running with VirtualBox

Before you start, make sure you have the lastest version of VirtualBox installed.

  1. Start VirtualBox and choose Import Appliance from the File menu. Select the downloaded file.
  2. Once VirtualBox has imported the appliance, select the imported virtual machine from the list of virtual machines and click on Start.
Note: Troubleshooting version 2018-08-14, which is based on Ubuntu 18.04 LTS (Bionic) with Linux 4.15.0-20-generic, if the directory /home/vagrant/bin is missing, fix your VirtualBox setup or upgrade your Ubuntu system from 16.04 (Xenial) to 18.04 (Bionic).

Using the VM

  1. Once the instance has started up, log in on the console using the username vagrant and password vagrant. This account has sudo privileges -- if you would like to set a root password, you can do that via sudo.
  1. You can SSH into the machine via the port 2222 on localhost of the host OS. (e.g. ssh -p 2222 vagrant@localhost )
  2. To access the MusicBrainz web site and web service, go to: http://localhost:5000
  3. To access the Search Server web service, go to: http://localhost:8080

Resetting Docker Containers

This new VM is based on docker containers. The docker containers should start automagically at boot time, but sometimes they don't. If any of the commands listed below do not work or give you strange docker/container errors, reset the containers with this command from the VM:

bin/reset-containers

Then try your command again.

Tuning your VM

We recommend that you give your VM 4GB of ram and 2 CPU cores, if that is possible. The more RAM you give to the VM, the faster it is going to run. To change the memory settings, you will need to shut down the VM, change the settings and then re-start the VM.

Running Replication

This VM comes "replication ready". To enable replication, and have the database catch up with the latest replication packets, you will need to sign up for a Live Data Feed access token on the MetaBrainz web site. Then SSH into the machine as described above then:

bin/set-token <replication token>

To start replication, do this:

bin/replicate now

This will load all of the changes to the database since the VM update. Unlike the previous versions of this VM, this command will not give you any output as it applies replication packets. If you want to see the progress, open another terminal window to the VM and then:

bin/tail-replication-log

This will allow you to see the output of the replication packets being applied.

Automating Replication

To turn on background replication, run:

bin/replicate start

to turn it off:

bin/replicate stop

We recommend leaving replication off for the time being, until you've built search indexes for the VM.

Building search indexes

The VM comes with support to build search indexes. In order to build the indexes, log in to the account and then:

sudo bin/reindex

Depending on your machine, this may take quite a long time. We recommend that you leave this running overnight. After the indexes are complete, you should be able to carry out indexed searches in your VM.

Note: Troubleshooting version 2018-08-14, running `sudo` as above is a workaround for a bug in `bin/reindex` that can be fixed by running once:
sudo sed -i '/sudo/!s/ls\|rm/sudo &/' bin/reindex

Accessing the database

To access the main postgres database from the outside, first run once (from the VM):

bin/turn-port db on
Note: Troubleshooting version 2018-08-14: Recreate the db container after running turn-port by running (from the VM):
docker-compose -f ~/musicbrainz/musicbrainz-docker/docker-compose.yml up -d db

then you can do this (from the host):

psql -h localhost -U musicbrainz -p 15432

You'll need to install the psql postgres command line client on your host OS.

By default, the password for the user musicbrainz is musicbrainz, as specified in the doc “Installing MusicBrainz Server”.

Accessing from a remote host

To access the MusicBrainz web site and web service from a remote host, run once (from the VM):

bin/set-web-server-name something.that.works

where something.that.works can be an IP address, or a host name, or a fully qualified domain name.

Then go to: http://something.that.works:5000

Note: Troubleshooting version 2018-08-14, the following warning message can safely be ignored
(node:80) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 12 resolve listeners added. Use emitter.setMaxListeners() to increase limit

Turning the VM into development box

If you need neither replication nor search, you may prefer to use the lighter virtual machine that contains sample data only.

MusicBrainz Sample Virtual Machine (Virtual Box)
Version: 2018-08-14
Size: 3.26GB
Format: Open Virtualization Archive (OVA)
Download via either BitTorrent or FTP
MD5: 5bd8714b59c32a7199e4d33446cceb78

If you would like to use the VM to do development instead of using it as a simple database slave, you'll need to edit lib/DBDefs.pm and set REPLICATION_TYPE to RT_STANDALONE and run admin/psql READWRITE and execute the following queries:

DELETE FROM annotation WHERE editor > (SELECT max(id) FROM editor);
DELETE FROM release_annotation WHERE NOT EXISTS (SELECT 1 FROM annotation WHERE annotation.id = release_annotation.annotation);

then from the command line execute:

admin/psql READWRITE < admin/sql/CreateFKConstraints.sql
admin/psql READWRITE < admin/sql/CreateFunctions.sql

Migrating the MusicBrainz Server from a Virtual Machine to a Physical Machine

Taken from AskUbuntu.

It is possible to migrate the MusicBrainz Server image (or any virtual image) from a virtual machine onto bare metal, should you desire to.

Requirements

  • A LiveCD or LiveUSB (this walkthrough will assume you are using a Ubuntu variant as that is what the image is based on.) If you are using an image from 2013-08-01 or earlier, you must have a Ubuntu 12.04.1 Live media.
  • USB media large enough to store the raw disk image for writing on a new computer.
  • Patience

Step 1: Converting the Disk to a Raw format

This step is included mostly for the sake of convenience. By doing some prep work on the image before writing, standard Linux tools can be used to write to and manipulate the disk afterwards. This can be done from your current host environment without booting to your LiveCD.

  • Load a terminal
cd /path/to/image/ 
# for VMWare
sudo apt-get install qemu-kvm
qemu-img convert your-vmware-disk.vmdk -O raw disk.img
# For VirtualBox
VBoxManage internalcommands converttoraw your-virtualbox-disk.vdi disk.img
  • Move disk.img to somewhere that you're not about to write to. If your target is another physical machine, then an external medium is probably the most convenient way to move the image. These instructions assume that you have moved it to /media/external/disk.img. Adjust accordingly.
  • Before you do anything permanent, make sure you do backups. These instructions will erase anything on the target disk or partition. It's better to be safe than sorry.

Option 1: Write the Image to it's Own Disk

This assumes that you're going to overwrite an entire disk. This is the most ideal situation since the server will have an entire disk to work with. If you're looking to have the image installed along side an existing operating system, skip to Option 2.

  • Boot into your LiveCD and Click Try Ubuntu.
  • Mount the volume where disk.img resides to somewhere safe (like /media/external).
  • Determine which volume is your target using a command like lsblk or blkid
  • Then we copy the contents of the image to the drive Replace sdX with the correct destination disk. The password for sudo is blank.
sudo dd if=/media/external/disk.img of=/dev/sdX
  • After this has finished, you can use gparted or another utility to manipulate the partition table to the desired layout. It is recommended to have a sizeable swap partition (see this article for some recommendations on how big it should be) as well as expand the root partition to use all remaining available space on the disk. If you desire a seperate boot partition for the system as well, set aside some space here. (Steps for this can be found in the Ubuntu community documentation)
  • If you are using an image from 2013-08-01 or earlier, please see here.

Option 2: Installing the Image Alongside Another OS

This is potentially safer than the above method and is very similar. Your LiveCD must be the same Ubuntu version as the virtual image.

  • Boot into the LiveCD and click Install Ubuntu.
  • Follow the installer prompts, repartition things as you see fit. When the installer finishes, it will ask you to reboot. Do not reboot. You must be in the Live environment for the next instructions.
  • Mount your new Ubuntu install (target) and your external disk where you stored disk.img earlier.
  • Mount disk.img.
sudo mkdir /media/oldinstall
sudo mount -o loop /media/external/disk.img /media/oldinstall
  • From there you can either cherry pick files or copy over everything on top of your target. For example, using rsync:
sudo rsync /media/oldinstall /media/newinstall

Changing Kernel Versions

Parts taken from the Ubuntu community documentation.

For images from 2013-08-01 and older, you will need to change the kernel flavour from -virtual to -generic. The -virtual kernel flavour is lighter, but only supports hardware typically seen in virtual environments while -generic provides support for a wider range of hardware, such as what you would see in a physical server. Both of these versions are provided in the default Ubuntu repositories.

  • Boot into your LiveCD. Click "Try Ubuntu"
  • Open a terminal.
  • Determine which disk is yours using a command like sudo lsblk or sudo blkid.
  • Mount your root partition. X is the drive letter and Y is the partition number:
sudo mount /dev/sdXY /mnt
    • If you have a seperate boot partition, mount it as well (Z is the boot partition number):
sudo mount /dev/sdXZ /mnt/boot
  • Mount the virtual filesystems using the following single command:
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
  • Chroot into your installation
sudo chroot /mnt
  • Install the latest -generic image and headers by installing the metapackage. This will regenerate GRUB's configuration.
sudo apt-get install linux-image-generic linux-header-generic
  • Purge the -virtual image and headers. This should also regenerate GRUB's configuration:
sudo apt-get purge --auto-remove linux-image-virtual linux-header-virtual
  • Check for leftover -virtual images in /boot. If you find any, remove them and then run sudo update-grub
  • Reboot into your installation.

Setup MusicBrainz Server from source code

This can potentially be a very laborious and time consuming method of getting a functioning MusicBrainz server. Using the virtual machine is recommended, except for development purpose.

Get a copy of musicbrainz-server from git:

git clone --recursive https://github.com/metabrainz/musicbrainz-server.git musicbrainz-server
cd musicbrainz-server

And follow the instructions in the INSTALL file.

Support

The setup process may look daunting, but please don't let this discourage you; the INSTALL is thorough and contains a lot of information, and we are willing to provide assistance. If you have questions about installing, join us in the #metabrainz IRC channel or post a question on the community forum and we will attempt to help you out.

We recommend that you dive in and give it a try - who knows how far you'll get and what you might learn along the way!

Requirements

In order to set up a running MusicBrainz server with the full database you will need:

  • A linux box, preferably Ubuntu.
  • 20GB+ of free disk space, (if you are a developer and only want the server code and database structure somewhat less should do).
  • Git knowledge which will enable you to check out the source code.
  • PostgreSQL 9.5 or later (and some other dependencies)

As a developer the following knowledge/skills are beneficial:

  • Perl and a number of perl modules, PostgreSQL, nginx.
  • How to compile and install packages from source on a Linux box.
  • How to patch existing packages, although we can help you out if you have questions about that.

Note: The server has never been ported to Windows, and we suspect that it would be a fair amount of work to make that happen.

External Links