User:Derat/MusicBrainzServerDevelopmentNotes

Here are some personal notes about trying to do MusicBrainz server development, starting in June 2023.

Getting the code and setting up my local environment

I cloned the musicbrainz-server and musicbrainz-docker repositories.

Next, I followed the instructions in the "Local development of MusicBrainz Server" section of musicbrainz-docker's README.md. I encountered a DateTime::Locale version error the first time I ran the createdb.sh command, but it worked when I ran it again.

I stepped away from the code for a few weeks at this point. When I came back, I synced both repositories and ran the following command to recreate the database:

sudo docker-compose run --rm musicbrainz recreatedb.sh -sample -fetch

I ran the following to start the containers (without detaching from the terminal, so I could watch the logs):

sudo docker-compose up

At this point, I was able to access my local server at http://localhost:5000/.

The server doesn't seem to have search indexes by default (I think there are more steps for getting that working), so I had to look up an MBID on the real MB site: http://localhost:5000/release/15f5c2cb-41ea-3630-a0f6-2f86785cf608

I edited root/release/add_cover_art.tt in the musicbrainz-server repo and verified that my changes showed up at http://localhost:5000/release/15f5c2cb-41ea-3630-a0f6-2f86785cf608/add-cover-art.

Creating an account

I added a user account via the web interface but my server wasn't able to send email messages, so I wasn't able to verify my email address (needed to perform edits).

I ran the following to connect to PostgreSQL:

sudo docker-compose run musicbrainz psql postgres -U musicbrainz -h db

Next, I ran the following to manually mark my account as being verified:

\l                -- list databases
\c musicbrainz_db -- change DB
\dt               -- list tables
UPDATE EDITOR SET email = 'user@example.org', email_confirm_date = now() WHERE name = 'my_username';

At this point, the server viewed my account as having been verified.

The following will set the account to be an auto-editor:

sudo docker-compose exec db psql -d musicbrainz_db -U musicbrainz -c "UPDATE editor SET privs = privs | 1 WHERE name = 'my_username'"

Linting code

I run the following to lint JavaScript changes:

./node_modules/.bin/eslint $(git diff --name-only master | grep '\.js$')

This will type-check JavaScript files:

./node_modules/.bin/flow

I got a few persistent Cannot resolve module `../../typeInfo.js`. errors unrelated to my changes. This also caused the full set of unit tests to not run. I finally fixed that by running

sudo docker-compose run musicbrainz script/create_test_db.sh

and then

sudo docker-compose run musicbrainz script/compile_resources.sh test

again. See this discussion for more info.

Running tests

JavaScript unit tests can be run by visiting http://localhost:5000/static/scripts/tests/web.html. I had trouble getting resources to be rebuild so that a new test file would be executed (after adding it to root/static/scripts/tests/index-web.js); I think that running this might've done the trick:

sudo docker-compose run --rm musicbrainz script/compile_resources.sh

I didn't have any luck running Selenium tests against my local installation when following the instructions from the "Selenium" section of musicbrainz-server's HACKING.md. The setup commands seemed to complete successfully:

sudo docker-compose run --rm musicbrainz ./script/create_test_db.sh SELENIUM
sudo docker-compose run --rm musicbrainz ./script/compile_resources.sh

Running ./t/selenium.js --help outside of Docker just produced an error, though:

node:fs:1396
  handleErrorFromBinding(ctx);
  ^

Error: ENOENT: no such file or directory, mkdir '/musicbrainz-server/.nyc_output'
    at Object.mkdirSync (node:fs:1396:3)
    at file:///home/.../temp/src/musicbrainz-server/root/utility/writeCoverage.mjs:18:6
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25) {
  errno: -2,
  syscall: 'mkdir',
  code: 'ENOENT',
  path: '/musicbrainz-server/.nyc_output'
}

Node.js v18.16.1

I couldn't figure out where MB_SERVER_ROOT was getting set incorrectly, and I didn't expect the test script to work within Docker (since I assume it'd have no way to access Chrome), so I just relied on Jenkins to run my tests after I uploaded a PR.

Enabling indexed searches

This was just a matter of running the following in my musicbrainz-docker checkout:

sudo docker-compose exec indexer python -m sir reindex

It died after several hours while building the recordings index (my system only had ~6.6 GB of RAM available), but I was just interested in testing artist search, so that was good enough for me.

Exercising cover art code

I was working on a change to the add-cover-art form, and I had trouble figuring out how to exercise it manually. HACKING-CAA.md describes how to emulate the Cover Art Archive locally, but I couldn't get it to work in conjunction with musicbrainz-docker. I eventually just hacked the code to report success without uploading to the CAA.

I commented out a bunch of code in lib/MusicBrainz/Server/Controller/WS/js.pm by adding =begin and =cut lines:

sub cover_art_upload : Chained('root') PathPart('cover-art-upload') Args(1)
{
    // ...
    my $context = $c->model('MB')->context;

=begin
    unless ($c->model('CoverArtArchive')->exists_for_release_gid($gid)) {
    // ...
            $self->_detach_with_ia_server_error($c, $response->decoded_content);
        }
    }
=cut

    my $id = $c->request->params->{image_id} // $c->model('CoverArtArchive')->fresh_id;
    // ...

I also updated root/static/scripts/edit/MB/CoverArt.js to return early without actually uploading images:

MB.CoverArt.upload_image = function (postfields, file) {
  var deferred = $.Deferred();

  window.setTimeout(() => {
    deferred.notify(50);
  }, 500);
  window.setTimeout(() => {
    deferred.notify(100);
    deferred.resolve();
  }, 1000);
  return deferred.promise();

  var formdata = new window.FormData();
  // ...

Rebuilding PostgreSQL indexes

At some point, I started seeing a bunch of warnings like these in the musicbrainz-docker output:

db_1           | 2024-04-22 15:58:36.679 UTC [244] WARNING:  collation "musicbrainz" has version mismatch
db_1           | 2024-04-22 15:58:36.679 UTC [244] DETAIL:  The collation in the database was created using version 153.14, but the operating system provides version 153.120.
db_1           | 2024-04-22 15:58:36.679 UTC [244] HINT:  Rebuild all objects affected by this collation and run ALTER COLLATION musicbrainz.musicbrainz REFRESH VERSION, or build PostgreSQL with the right library version.

The fix was to run the script added by #3062 for MBS-13336 as described in MAINTENANCE.md:

sudo docker-compose run musicbrainz .admin/RebuildIndexesUsingCollations.pl --noconcurrently