Redis: Persistence Power

Redis: Persistence Power
Nick Quaranto / @qrush / nick@quaran.to
Tuesday, August 10, 2010

What is Redis?
“advanced key-value store”
REmote DIctionary Server
data structures server

YOUR APP

YOUR APP
REDIS

the basics
persist data as you think of it
in memory, sync to disk in background
ridiculously fast
master-slave replication
keys = strings, value = data structures

http://try.redis-db.com

use it: redis-cli
% ./redis-cli SET user:1:name qrush

OK
% ./redis-cli GET user:1:name

"qrush"

use it: redis-rb
% gem install redis
% irb -rubygems -rredis
>> $redis = Redis.new

=> #<Redis client v2.0.3 connected...
>> $redis.set "user:1:name", "qrush"

=> "OK"
>> $redis.get "user:1:name"

=> "qrush"
FEATURE
SWITCHES
STRINGS
based on
http://github.com/blog/677
http://github.com/bvandenbos/redis_feature_control
Rediswitch.features << :super_secret
Rediswitch.features << :payment_gateway
Rediswitch.features << :twitter
if Rediswitch.enabled?(:twitter)
# post to twitter
else
# failwhale ahoy!
end

begin
# take some money
rescue PaymentGateway::TotallyDown => ohno
Rediswitch.disable(:payment_gateway)
# notify the troops
end

class Rediswitch
def self.enabled?(feature)
$redis.exists(feature)
end
def self.enable(feature)
$redis.incr(feature)
end
def self.disable(feature)
$redis.del(feature)
end
end

feature switch lessons
the real win: no-deploy configuration
fast enough to be transparent
next step: separate users into buckets with sets
http://github.com/jamesgolick/rollout

RATE
LIMITER
STRINGS
soon to be in place at
http://hoptoadapp.com
class Choker
def restrict?
track
count_for > 60
end
end

class Choker
def count_for
$memcache.get(key, true).to_i
end
end

class Choker
def track
if !$memcache.get(key, true)
$memcache.add(key, "0", 1.minute.from_now, true)
end
$memcache.incr(key)
end
end

class Choker
def track
if !$redis.exists(key)
$redis.setex(key, 60, 0)
end
$redis.incr(key)
end
end

class Choker
def count_for
$redis.get(key).to_i
end
end

rate limiter lessons
expire semantics are changing in redis 2.2
benchmark the crap out of it
could use a sorted set instead of strings

API USAGE
LOGGING
STRINGS
SORTED SETS
based off
http://www.production-hacks.com/2010/07/10/redis-api-access-logger/

# one way to do it
class ActionHit < ActiveRecord::Base
# t.string :controller
# t.string :action
# t.integer :counter
end
class UserHit < ActiveRecord::Base

# t.string :controller_action
# t.integer :user_id
# t.integer :counter
end

# for all controllers
{
"statuses#update" => 1410,
"users#create" => 931,
"home#index" => 2936
}
# users hitting an action

{
"101" => 42,
"102" => 13,
"103" => 34
}

class StatusesController < ApplicationController
def update
$redis.incr "statuses#update"
$redis.incr "statuses#update:#{user.id}"
end
end

class StatusesController < ApplicationController
def update
key = "statuses#update"
$redis.zincrby "actions", 1, key
$redis.zincrby "users:#{key}", 1, user.id
end
end

# hits for a specific user
>> $redis.zscore "users:statuses#update", 1001
=> 42
# list all the controller actions, sorted

>> $redis.zrevrange "actions", 0, -1,
:with_scores => true
=> ["home#index", "2936",

"statuses#update", "1410",
"users#create", "931"]

api usage logging
lessons
sorted set = high score list
bad at historical usage, trends
good for a simple heartbeat or pulse

JOB QUEUE
LISTS
based on
http://github.com/defunkt/resque
class Staple
@queue = :default
def self.perform(post_id, tempfile)

# complex image resizing, cropping
end
end

class Post < ActiveRecord::Base
after_save :process_with_stapler
def process_with_stapler
Resque.enqueue(Staple, self.id, @tempfile)
end
end

module Resque
extend self
def push(queue, item)

redis.rpush "q:#{queue}", encode(item)
end
def pop(queue)
decode redis.lpop("q:#{queue}")
end
end

class Resque::Worker
def work
loop do
if job = Resque.pop(queue)
job.perform
else
sleep 5
end
end
end
end

module Resque
extend self
def bpop(queue)
decode redis.blpop("q:#{queue}")
end
end

class Resque::Worker
def work
loop do
job = Resque.bpop(queue)
job.perform
end
end
end

job queue lessons
guaranteed atomic actions, no row locking
blocking commands simplify daemons
many more queue commands in redis itself!

GLOBAL
ERRORS
SETS
MULTI/EXEC
a new feature at
http://hoptoadapp.com
# text :globals, :default => '', :null => false
class Project < ActiveRecord::Base

def has_global?(name)
@globals ||= globals.gsub(/,/,' ').split
@globals.include?(name)
end
end

# MORE TABLES!!!!
class Global < ActiveRecord::Base

belongs_to :project
end

has_many :globals
end

# Project#global_errors
["MySQL::Error",
"MemCache::Error",
"Net::HTTPFatalError"]

def global_key
"project-#{id}-globals"
end
def has_global?(name)
$redis.sismember(global_key, name)
end
end

after_save :save_globals
def save_globals
$redis.del global_key
@globals.each do |g|
$redis.sadd global_key, g
end
end
end

SISMEMBER SISMEMBER
[Mysql::Error, MemCache::Error, Net::HTTPFatalError]

DEL
[]

DEL
SADD
[Mysql::Error]

DEL
SADD
SADD
[Mysql::Error, OpenURI::HTTPError]

DEL
SISMEMBER
SADD
SADD
[Mysql::Error, OpenURI::HTTPError] []

MULTI
DEL
SADD
SISMEMBER
SADD
EXEC
[Mysql::Error, OpenURI::HTTPError]

after_save :save_globals
def save_globals
$redis.multi do
$redis.del global_key
@globals.each do |g|
$redis.sadd global_key, g
end
end
end
end

global error lessons
avoid joins for simple data
consider race conditions
use append-only file (AOF)

MULTIPLAYER
NOTEPAD
PUB/SUB
based on
http://github.com/laktek/realie
# usage: ruby pub.rb room username
data = {"user" => ARGV[1]}

loop do
msg = STDIN.gets
$redis.publish ARGV[0],
data.merge('msg' => msg.strip).to_json
end

# sub.rb
$redis = Redis.new(:timeout => 0)

$redis.subscribe('rubyonrails', 'rubymidwest') do |on|
on.message do |room, msg|
data = JSON.parse(msg)
puts "##{room} - [#{data['user']}]: #{data['msg']}"
end
end

% ruby pub.rb rubymidwest qrush
i give up, i hate markdown
% ruby sub.rb
#rubymidwest - [qrush]: i give up, i hate markdown
#rubyonrails - [railsn00b]: undefined method posts_path? wtf?
#rubymidwest - [turbage]: seriously.

multiplayer notepad
lessons
combine with other data structures
can subscribe to channels via patterns
concurrency in ruby is hard
use eventmachine! (or node.js)

more to learn
know your data! (via @antirez)
command reference on the wiki
active IRC, mailing list

AKASENTAI.com
redis in the cloud
Thanks!
http://redis.io @qrush
http://rediscookbook.com
http://scr.bi/redispower
BONUS ROUND!
I prepared way too many examples. Jackpot!

URL
SHORTENER
STRINGS
based on
http://github.com/mattmatt/relink
require 'sinatra'
require 'redis_url'
post '/' do
RedisUrl.new(params[:url]).save
end

class RedisUrl
attr_accessor :url, :id
def initialize(url)
@url = url
@id = seed # unique string algorithm
end
def save
$redis.set("relink.url|#{@id}", @url)
$redis.set("relink.url.rev|#{@url}", @id)
end
end

get %r{/(.+)} do |url|
u = RedisUrl.find(url)
if u
u.clicked
redirect u.url
else
status 404
end
end

class RedisUrl
def self.find(id)
u = $redis.get("relink.url|#{id}")
if u
redis_url = RedisUrl.new(u)
redis_url.id = id
redis_url
end
end
def clicked
$redis.incr("relink.url.clicks|#{@id}")
end
end

url shortener lessons
common pattern: namespacing
incr/decr assumes value is an integer
wrap behavior into ActiveRecord-like objects
next step: store URLs in a list

LIVE
DEBUGGING
LISTS
based on
http://github.com/quirkey/redisk
def after_save
begin
# make request to external service
rescue Exception => ex
logger.error "this shouldn't ever happen!"
logger.error ex
logger.error ex.backtrace
end
end

# config/initializers/logger.rb
require 'redisk'
path = "#{Rails.env}.log"
config.logger = Redisk::Logger.new(path)

class Redisk::IO
def write(string)
redis.rpush "#{name}:_list", string
end
def self.readlines(name)
redis.lrange("#{name}:_list", 0, -1)
end
end

live debugging lessons
enables real-time data about your system
dump serialized/marshalled data fast
run the redis instance on a different box
dive deeper: hummingbird

COUNTING
DOWNLOADS
STRINGS
SORTED SETS
HASHES
based on
http://github.com/rubygems/gemcutter
# bad idea, dude
class Download < ActiveRecord::Base

belongs_to :rubygem
end
class Rubygem < ActiveRecord::Base

has_many :downloads
end

class Download
def self.incr(rubygem)
$redis.incr("all")
$redis.incr(rubygem)
$redis.zincrby("today", 1, rubygem)
end
end

class Download
def self.rollover(version)
$redis.rename "today", "yesterday"
dls = Hash[*$redis.zrange("yesterday",
0, -1,
:with_scores => true)]
dls.each do |key, score|

$redis.hincrby key, Date.today, score
Rubygem.find_by_name(key).increment!(:downloads, score)
end
end
end

get "/api/v1/downloads/rails.json" do
$redis.hgetall("rails").to_json
end
# returns...
{
"2010-07-09" => 1908,
"2010-07-10" => 1032,
"2010-07-11" => 1091,
}

counting downloads
lessons
hybrid approach does work!
redis is really not for search
test your migration away from SQL

Redis: Persistence Power

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Redis: Persistence Power

Uploaded by

Copyright:

Available Formats

Redis: Persistence Power

Nick Quaranto / @qrush / nick@quaran.to

Tuesday, August 10, 2010

“advanced key-value store”

REmote DIctionary Server

data structures server

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

persist data as you think of it

in memory, sync to disk in background

keys = strings, value = data structures

Tuesday, August 10, 2010

Tuesday, August 10, 2010

% ./redis-cli SET user:1:name qrush

% ./redis-cli GET user:1:name

Tuesday, August 10, 2010

>> $redis = Redis.new

>> $redis.set "user:1:name", "qrush"

>> $redis.get "user:1:name"

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

the real win: no-deploy configuration

fast enough to be transparent

next step: separate users into buckets with sets

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

expire semantics are changing in redis 2.2

benchmark the crap out of it

could use a sorted set instead of strings

Tuesday, August 10, 2010

Tuesday, August 10, 2010

class UserHit < ActiveRecord::Base

Tuesday, August 10, 2010

# users hitting an action

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

# list all the controller actions, sorted

=> ["home#index", "2936",

Tuesday, August 10, 2010

sorted set = high score list

bad at historical usage, trends

good for a simple heartbeat or pulse

Tuesday, August 10, 2010

def self.perform(post_id, tempfile)

Tuesday, August 10, 2010

Tuesday, August 10, 2010

def push(queue, item)

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

Tuesday, August 10, 2010

guaranteed atomic actions, no row locking

blocking commands simplify daemons

many more queue commands in redis itself!

Tuesday, August 10, 2010

class Project < ActiveRecord::Base

Tuesday, August 10, 2010