Difference between revisions of "User:Jokipii"

From MusicBrainz Wiki
(status update)
Line 46: Line 46:
 
* Start using [https://github.com/philipmat/discogs-xml2db discogs-xml2db] to produce Discogs database [[Image:Checkmark.png]]'''''Done'''''
 
* Start using [https://github.com/philipmat/discogs-xml2db discogs-xml2db] to produce Discogs database [[Image:Checkmark.png]]'''''Done'''''
 
* Better documentation
 
* Better documentation
* Map Discogs credits <-> MB Advanced relationships
+
** [[User:Jokipii/Discogs mapping]] documentation how bot maps values between Discogs and MB '''''Need some updates'''''
 +
* Map Discogs credits <-> MB Advanced relationships '''''Started'''''
  
 
== Some stats ==
 
== Some stats ==

Revision as of 10:57, 12 July 2012

Antti Jokipii [MB: Jokipii and operator of Jokipii_bot | IRC: Jokipii | Wiki: Jokipii | Last.fm AnttiJokipii]

I am currently trying to improve linking between MusicBrainz and Discogs. I have both databases installed on PostgreSQL. Bot code can be found at musicbrainz-bot and code that produces Discogs database from monthly XML dumps found at discogs-xml2db.

Userscripts

Here is userscript that makes voting for Discogs links easier.

Bot queue

Set descriptions and number of links

In Progress

Artist Discogs links based on name match and other evidence (linked releases, linked release-groups, same external urls, same release names, "member of band"-link, "is person"-link, VA release track artist)
* ~ 40000
Label links based on name match and catalog numbers
* ~ 10000
Release links identified by match on normalized catalog number, release name, linked label, format, same number of tracks, same release country, same release year.
* ~ 1000
Advanced relationshipd between recordings and artists where both are already linked to Discogs
* remixer ~ 40000
Barcodes from linked Discogs releases
* 500

Not Started

Artist "is person"-links based on Discogs data
* 1700
Artist "member of band"-links based on Discogs data
* 23330
Advanced relationships between releases and artists where both are already linked to Discogs
* Producer Hand made example 74164
* Mastered 27970
* and certainly lots also in other relationship classes

Done

Artist Discogs links with name match. Have release(s) with Discogs links. All releases found point to same artist at Discogs.
Artist types based on disambiguation comment
Artist country based on disambiguation comment
Artist country based on Discogs profile text
Release links identified by exact match on catalog number, release name, linked label, format, same number of tracks and same release country.
Release links identified by match on normalized catalog number, release name, linked label, format, same number of tracks, same release country, same release year and same barcode.
Removing dead (404 error) Discogs artist and label links

Bot programming tasks

Some stats

MusicBrainz Total Discogs Total Links (all these are not unique) Percent done (compared to smaller total) Sum of unique MusicBrainz releases connected to linked entities Percent of all MusicBrainz releases Sum of unique Discogs releases connected to linked entities Percent of all Discogs releases
Releases: 988301 2720810 171170 17%
Release groups: 822442 365081 47387 13% 106445 11% 277207 10%
Artist: 626598 2100250 110825 18% 606737 61% 1738474 64%
Label: 55844 245988 16004 29% 414436 42% see note 1542803 57%

note: In MB only 567392 releases have label information, and 420909 don't have.


2012-02-23 MusicBrainz Total Discogs Total Links (all these are not unique) Percent done (compared to smaller total)
Releases: 1008061 2926422 182581 18%
Release Groups: 839314 405891 73656 18%
Artists: 644784 2251519 126210 20%
Labels: 58038 300452 17141 30%


2012-03-23 MusicBrainz Total Discogs Total Links (all these are not unique) Percent done (compared to smaller total)
Releases: 1016640 2926422 200576 20%
Release Groups: 846637 405891 80482 20%
Artists: 651243 2251526 143468 22%
Labels: 58872 300452 17322 29%


2012-04-23 MusicBrainz Total Discogs Total Links (all these are not unique) Percent done (compared to smaller total)
Releases: 1027422 3045567 225871 22%
Release Groups: 854785 423610 90931 21%
Artists: 658827 2328770 149744 23%
Labels: 59775 323996 17529 29%


2012-05-23 MusicBrainz Total Discogs Total Links (all these are not unique) Percent done (compared to smaller total)
Releases: 1035849 3045567 229590 22%
Release Groups: 860665 423610 94024 22%
Artists: 664330 2328770 164656 25%
Labels: 60456 323996 17949 30%


2012-06-23 MusicBrainz Total Discogs Total Links (all these are not unique) Percent done (compared to smaller total)
Releases: 1044483 3166200 239847 23%
Release Groups: 867139 441382 96020 22%
Artists: 669934 2407765 183851 27%
Labels: 61057 347133 19938 33%