Development/Summer of Code/2017: Difference between revisions

From MusicBrainz Wiki
Jump to navigationJump to search
Line 8: Line 8:


===BigQuery upload and statistics===
===BigQuery upload and statistics===

Skills: Python, design

We have some prototype code to upload the contents of AcousticBrainz to google bigquery.
We would like to finish this project so that the data is available in bigquery. This involves:
# Finishing the upload tool, fixing any existing issues
# Check that the data that we have stored in bigquery is in a format apt for searching. The value that BQ brings for us is to be able to quickly search through all of our data quickly. We need to make sure that it's possible. This could involve surveys/emails with other in the musicbrainz/Music research community
# Once the data is uploaded, create an interactive statistics page for the AcousticBrainz website, showing images similar to our original blog post


===New machine learning infrastructure===
===New machine learning infrastructure===

Revision as of 21:09, 8 February 2017

This page captures our ideas for Google Summer of Code projects for 2017:

MusicBrainz

Bitmap: Please add ideas!

AcousticBrainz

BigQuery upload and statistics

Skills: Python, design

We have some prototype code to upload the contents of AcousticBrainz to google bigquery. We would like to finish this project so that the data is available in bigquery. This involves:

  1. Finishing the upload tool, fixing any existing issues
  2. Check that the data that we have stored in bigquery is in a format apt for searching. The value that BQ brings for us is to be able to quickly search through all of our data quickly. We need to make sure that it's possible. This could involve surveys/emails with other in the musicbrainz/Music research community
  3. Once the data is uploaded, create an interactive statistics page for the AcousticBrainz website, showing images similar to our original blog post

New machine learning infrastructure

Storage for detailed analysis files

CritiqueBrainz

Direct access to MusicBrainz database

Proposed mentor: Gentlecat
Languages/skills: Python, Flask, SQL (PostgreSQL, SQLAlchemy), Docker, Consul

So far, the biggest cause for slowdown in CritiqueBrainz are requests to MusicBrainz web service. It's not that MusicBrainz WS is slow, it's just that some pages on CritiqueBrainz require a lot of MusicBrainz data, which might take a very long time to retrieve. This can be caused by the complexity of a request, or by a number of them (when showing multiple items, since there's no way to do batch-requests).

New infrastructure allows us to easily read data directly from the MusicBrainz database. Doing this in CritiqueBrainz will probably be a significant speedup.

See https://tickets.metabrainz.org/browse/CB-231.

Book reviews

Proposed mentor: Gentlecat
Languages/skills: Python, Flask, SQL (PostgreSQL, SQLAlchemy), JavaScript

Currently people can review release groups, events, and places. We'd like to expand that list by allowing people to publish reviews for books. Metadata about books can be retrieved from the BookBrainz project (by accessing the web API or its database directly). This project would require (1) extension of CritiqueBrainz back-end to support book reviews, and (2) changes to the web interface.

See https://tickets.metabrainz.org/browse/CB-240.

ListenBrainz

Create charts/graphs for user behaviour

Proposed mentors:mayhem, alastairp
Languages/skills: Python, Flask, BiqQuery, InfluxDB, data science, graphing, visualization, data architecture

ListenBrainz is preparing to stream its listen data to Big Query where anyone can have access to it in real time. From this data that is stored in BigQuery we wish to have a student build a general charting/graphing system that allows future contributors to explore the data with BigQuery. Any user should be able to craft a query that can be turned into a graph/visualization on the ListenBrainz site, with minimal effort. If a user crafts an interesting query, they should be able to open a pull request and supply the details of the query in order for the LB team to add this graph to the site.

This project requires building the behind the scenes BigQuery access, caching, periodic updates and synchronization between the ListenBrainz server and the BigQuery data store.

A way to associate listens with MBIDs

Proposed mentors: ruaok, alastairp, gentlecat
Languages/skills: Python
Forum for discussion

Last.fm is broken because of the terrible way it handles metadata (artists with the same name are jumbled into a single page; at the same time, there are often multiple pages for the same artist/album/track due to spelling variations). ListenBrainz is smarter by taking advantage of MBIDs. But there needs to be some sort of interface for identifying listens as being for a particular track (or recording) MBID. This could allow the user to identify an album they listened to on Spotify as the same one they listen to in iTunes a few days later. Then they wouldn't remain separate artists or albums in the stats due to differences in metadata alone.

BookBrainz

Data Importing

Proposed mentors: LordSputnik or Leftmost
Languages/skills: Browser JS, Node.js or Python, SQL/PostgreSQL
Forum for discussion

At last year's summit, the two BookBrainz lead developers, Leftmost and LordSputnik worked on a plan for importing third party data into BookBrainz. This plan has several stages. First, data sources need to be identified, including mass import sources with freely available data, such as libraries, and manual import sources, such as online book stores and other user-contributed databases. The next stage is to update the database to introduce an "Import" object, which can be used to distinguish mass imported data from (usually better quality) user contributions. Then, actual import bots for mass import and userscripts for manual import will need to be written. Finally, it would desirable (but not necessary if time is short) to introduce an interface to the BookBrainz site to allow users to review automatically imported data, and approve it.

Web API

Proposed Mentors: LordSputnik/Leftmost
Languages/skills: Node.js, ES6, Python, Redis, OAuth
Forum for discussion

We’re currently in the process of switching to using Node.js for all server side code. As part of this, our schema has been redesigned, and the current Python-based web API will no longer work.

We'd like a new and improved JSON web API to be designed and implemented. The design would clearly describe the result of each different query to the web API, and give examples of output. It would also describe the workings of any additional features to be implemented - for example, authentication, caching and rate limiting. Authentication in the web API is a particular challenge, since the current MB OAuth setup requires a GUI.

The web API should be written using the koa.js Node.js server framework, so that the resulting code is as clean and minimal as possible. Tests should be written in parallel with the implementation, adapting and expanding on the tests for the existing Python web API. The priority for this task is a solid plan and quality code, not a complete implementation (although that would be nice!)

MusicBrainz Picard

Qt5/Py3 Port

Proposed Mentors: zas/bitmap
Languages/skills: Python, PyQt5

Picard is going to be receiving a complete make-over code/feature/UI wise for v2.0. A part of the v2.0 plans is porting the existing code to PyQt5/Py3 from PyQt4/py2 to allow for future support and better encoding compatibility along with new features of Qt5(such as High DPI scaling, better Widget and networking support and better cross-platform compatibility) and along with all the bug fixes that come with PyQt5 which will help cleaning up the code-base for Picard.

For more information see: PICARD-588 PICARD-960

Picard Plugins API v2

Proposed Mentors: zas/bitmap
Languages/skills: Python, PyQt5, Flask

Picard 2.0 is going to be PyQt5 based, which means all the existing plugins have to be ported to PyQt5/Py3. Since Picard v1 plugins will be incompatible with v2 and vice versa, we need to implement a new v2 end-point for the Picard website. There is also a need of providing a uniform GUI api for plugins to allow quick GUI option settings, i18n support and UI uniformity with the rest of the Picard code. Also, additional metadata headers need to be implemented to allow cross-platform compatibility checks.

For more info see: PICARD-977