User:RobertKaye/3Scale Non Rate Limited Web Service

From MusicBrainz Wiki
< User:RobertKaye
Revision as of 17:16, 9 April 2012 by Djce (talk | contribs) (Link to MBH-222 for Varnish install)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Motivation, goals and measuring success

The motivations to create a non-rate limited web service are:

  • Allow us to keep track of commercial customers and charge for commercial access to our web service.
  • Give end-users the option to pay for non-rate limited access to our web service so they can tag their music collection faster

The end goal of this project is to create two tiered service of our ws/2 (not ws/1) web service:

  • where anonymous web service users get free service given the current limits.
  • where people who have signed up with 3scale and have credit in their account get access to our ws/2 web service with no required pauses between requests, but are limited to X concurrent requests.
  • where people who have signed up with 3scale, but have run out of credit in their account, simply revert back to the rate limited web service.

We will have succeeded if a developer can:

  • Start with an existing application that uses the MB web service version 2
  • Sign up for service at the MusicBrainz WS sign-up form hosted at 3scale
  • Add the application/developer keys to their application.
  • Have their application successfully make non-rate limited requests of the ws/2 web service.


The flow of requests through this improved service will look as follows:

3scale request flow.png

A request for ws/2 will arrive at nginx on carl/lenny and first be passed to the varnish caching web server. Varnish with its installed 3scale plug-in will validate a request and add a header to the request if the request has been authorized to use the non-rate limited web service. The request is then passed on from varnish back to the nginx load balancer that balances the web front ends (asterix, astro, etc). After the load balancer sends the request to a web front end, the front end will ask the rate limiter if the request should be honored. The rate limiter should then be able to identify the request as a 3scale approved request or a bog-standard rate limited request. The rate limiter should carry out its job as usual if the request is rate limited; if the request is 3scale approved, the rate limiter should check to make sure that no more than X concurrent requests are being handled for the same application key.

Front end & DNS

There will be no changes on how to talk to our web service. No new DNS entries will be needed; all the the communication will happen via extra request header fields.


The varnish web service will need to be installed on Carl/Lenny (hopefully from packages) and the 3scale plugin will need to be installed. ocharles will determine the initial configuration of the varnish server and its plugin.

The 3scale plugin will take care of making requests to 3scale to authenticate client/app keys. This plugin will make usage reports to 3scale and also see if the user is allowed to make more calls. This plugin should require only normal web access -- no firewall changes will be needed and no SSL communications will be required.

Rate limiter

The rate limiter will need to be modified to identify 3scale approved requests and apply concurrent limit check. The number of maximum allowable concurrent requests should be configurable. The exact mechanics of how the rate limiter will identify requests that are 3scale approved still remains to be determined. See 'breakdown of tasks' for more information on this.

The rate limiter should create a new application class that allows identified users a very high rate limit. For instance if we allow X concurrent requests we may allow X * Y requests in a 10 second window, where Y would be a high number such as 50 - 100.

3scale MusicBrainz developer portal

Navap and ruaok will setup the developer portal at 3scale and determine packages and pricing for the non-rate limited web service.

Breakdown of tasks

  • setup of the 3scale offerings (ruaok/navap)
  • creating the initial varnish config file (ocharles -- MBS-4421)
  • augment this specification to indicate which headers will be set by varnish and how to pass the needed client/app keys to the rate limiter (ocharles - MBS-4421)
  • setup of varnish on carl/lenny (djce - MBH-212)
  • improvements to rate limiter (djce - MBH-213)
  • nginx changes (djce - MBH-214)


(answers to these questions have been added to the document above)

  • Is is true that all requests still go to ?
    • Yes.
  • DNS "A" for still resolves to carl/lenny?
    • Yes. There will be not visible changes to the "outside" world.
  • Therefore Varnish lives on carl/lenny?
    • Yes.
  • Varnish will be installed from standard Ubuntu packages?
  • What's the nature of the comms between the Varnish module and 3scale? Firewall implications? Uses SSL?
    • For every X requests made for one api key, varnish will make one call to 3scale to report usage and to get an ok to serve more requests. There should be no firewall implications nor will it require SSL.
  • Writing a reliable concurrency-checker could be *hard*. It might not be the ratelimit-server's job.
    • Our thought was to simply create another rate limiting group and set a very high limit for that group. For instance if we allow X concurrent requests we may allow X * Y requests in a 10 second window, where Y would be a high number such as 50 - 100.
      • So just to confirm: we are going for "still rate limited, but with a higher limit, and no concurrency limit", yes? Plenty of places on this page refer to concurrency limits and non-rate-limited service, so if "yes" then those references will need to be updated.