User:RobertKaye/3Scale Non Rate Limited Web Service

From MusicBrainz Wiki

Motivation, goals and measuring success

The motivations to create a non-rate limited web service are:

  • Allow us to keep track of commercial customers and charge for commercial access to our web service.
  • Give end-users the option to pay for non-rate limited access to our web service so they can tag their music collection faster

The end goal of this project is to create two tiered service of our ws/2 (not ws/1) web service:

  • where anonymous web service users get free service given the current limits.
  • where people who have signed up with 3scale and have credit in their account get access to our ws/2 web service with no required pauses between requests, but are limited to X concurrent requests.
  • where people who have signed up with 3scale, but have run out of credit in their account, simply revert back to the rate limited web service.

We will have succeeded if a developer can:

  • Start with an existing application that uses the MB web service version 2
  • Sign up for service at the MusicBrainz WS sign-up form hosted at 3scale
  • Add the application/developer keys to their application.
  • Have their application successfully make non-rate limited requests of the ws/2 web service.

Design

The flow of requests through this improved service will look as follows:

3scale request flow.png

A request for ws/2 will arrive at nginx on carl/lenny and first be passed to the varnish caching web server. Varnish with its installed 3scale plug-in will validate a request and add a header to the request if the request has been authorized to use the non-rate limited web service. The request is then passed on from varnish back to the nginx load balancer that balances the web front ends (asterix, astro, etc). After the load balancer sends the request to a web front end, the front end will ask the rate limiter if the request should be honored. The rate limiter should then be able to identify the request as a 3scale approved request or a bog-standard rate limited request. The rate limiter should carry out its job as usual if the request is rate limited; if the request is 3scale approved, the rate limiter should check to make sure that no more than X concurrent requests are being handled for the same application key.

Varnish

The varnish web service will need to be installed on Carl/Lenny and the [plugin] will need to be installed. ocharles will determine the initial configuration of the varnish server and its plugin.

Rate limiter

The rate limiter will need to be modified to identify 3scale approved requests and apply concurrent limit check. The number of maximum allowable concurrent requests should be configurable. The exact mechanics of how the rate limiter will identify requests that are 3scale approved still remains to be determined. See 'breakdown of tasks' for more information on this.

3scale MusicBrainz developer portal

User:Navap and User:RobertKaye will setup the developer portal at 3scale and determine packages and pricing for the non-rate limited web service.

Breakdown of tasks

  • setup of the 3scale offerings (ruaok/navap)
  • creating the initial varnish config file (ocharles)
  • augment this specification to indicate which headers will be set by varnish and how to pass the needed client/app keys to the rate limiter (ocharles)
  • setup of varnish on carl/lenny (djce)
  • improvements to rate limiter (djce)
  • nginx changes (djce)

Questions

Sorry. Probably being dense.  :-) -- djce

* Is is true that all requests still go to http://musicbrainz.org/ws/2/... ?
* DNS "A" for musicbrainz.org still resolves to carl/lenny?
* Therefore Varnish lives on carl/lenny?
* Varnish will be installed from standard Ubuntu packages?
* What's the nature of the comms between the Varnish module and 3scale?  Firewall implications?  Uses SSL?
* Writing a reliable concurrency-checker could be *hard*.  It might not be the ratelimit-server's job.