Distributed Quota Management in an Ad Serving Environment

How Reduce Data Manages its budgets without exceeding ad spends http://reducedata.com

Problem
Managing Quotas / Budgets in An Ad Serving Environment is not a trivial task. Ad Campaign Budgets Across a Distributed infrastructure when not managed properly leads to excess spends.

Previous Solution
Sync Every 5 seconds Ad Server Ad Server Ad Server Cassandra Data Hbase based campaign reporting data

Kafka Pipe

Data processing system

The Issues
Simple Syncing of data across distributed data structure is easy but sticking within limits is tough. 5 second refresh not guaranteed to be upto date. Reporting delays caused spends to exceed. Server specic budget was not proportionately allocated, based on historical availability of trafc on that server. No clear mechanism of allocation of budget spends or reallocation (when servers go down).

Solution
!

Our Solution was based on this white paper titled: Quota enforcement for high-performance distributed storage systems http://www.ssrc.ucsc.edu/Papers/pollackmsst07.pdf

Key Concepts
! !

Epoc : A day is divided into epocs. At each epoc DSPServers request for quota from Quota Management Cluster. Ad Server doesn't decide how much quota it needs instead a new entity the Quota Management Cluster decides how much quota to assign to it. Traffic for a Campaign maybe available at multiple endpoints(US East, US West, Asia)

How Quotas are Decided

!

Quota is distributed to end points (different data centre locations) and to the individual servers depending on the historical traffic / consumption of budgets. Two factors that decide how much is allocated in each epoc:
!

Pacing variable as decided by the advertiser Historical consumption data (for example has the campaign been spending lesser than planned?, if so, increase EPOC allocation)

Quota Management Cluster

!

Quota management cluster has multiple servers and needs a data store which supports transactions. The Quota Management Cluster is Synchronized using Zookeper.

Logical Architecture
Sync at every EPOC 2 ways Ad Server Ad Server Ad Server Quota Quota Management Management Server Server Cassandra based Cassandra Quota data based Quota data

Ad servers can sync with the Quota management server at EPOC which could be say every hour. Ad servers refund unspent balance to Quota Management Cluster. Quota management can be in one location initially but can be made redundant to each data centre. Quota Management Cluster uses a shared data store on cassandra to sync its other nodes for allocation.

Zookeeper Zookeeper

Sync Process
Quota Management Server Ad Server Zookeeper Cassandra Sync at every EPOC Push budgets, get unspent amounts Sync allocation data to cassandra cluster Check Server status Get list of available ad servers Push available budgets on a restart. Calculate available spends

Handling Ad Server Shut Downs

!

Zookeeper tracks server failures. Quota management system reallocates remaining budget (last budget allocated - amount spent) to a new or the same server when it restarts.

Handling Communications
The Quota Management Servers talk to Ad Servers using REST Calls. This required lowest development time and was built on top of the existing stack. Alternatives we considered but decided against. 2. Kafka 3. RPC over protobuf

Get in Touch
Engineering Team Reduce Data reducedata.com twitter: @reducedata facebook.com/reducedata engineering@reducedata.com