Sidekiq in Rails

sidekiq
Image modify from whitepages

http://sidekiq.org/
https://github.com/mperham/sidekiq
http://railscasts.com/episodes/366-sidekiq
[saɪdkeɪk] 应该能帮到一些人吧,哈哈。

阅读前需要了解Redis、Background Job等概念,本文自动化部署使用的是Capistrano 3。

Rails中配置Sidekiq

1
2
3
4
5
6
$ gem install sidekiq
# Installing connection_pool 2.0.0
# Installing redis 3.0.7
# Installing redis-namespace 1.4.1
# Installing sidekiq 3.1.3

将Redis的配置信息放到Rails 4中的secrets.yml中:

config/secrets.yml
1
2
3
4
5
6
7
8
9
redis: &redis
redis_server: 'localhost'
redis_port: 6379
redis_db_num: 0
redis_namespace: 'highlander_sidekiq'
development:
<<: *redis

initializers下新建sidekiq.rb文件,用来初始化Redis和Sidekiq config

initializers/sidekiq.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
redis_server = Rails.application.secrets.redis_server
redis_port = Rails.application.secrets.redis_port
redis_db_num = Rails.application.secrets.redis_db_num
redis_namespace = Rails.application.secrets.redis_namespace
Sidekiq.configure_server do |config|
p redis_server
config.redis = { url: "redis://#{redis_server}:#{redis_port}/#{redis_db_num}", namespace: redis_namespace }
end
Sidekiq.configure_client do |config|
config.redis = { url: "redis://#{redis_server}:#{redis_port}/#{redis_db_num}", namespace: redis_namespace }
end

sidekiq 配置参数

命令后加上 --help 可以看到其配置参数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ bundle exec sidekiq --help
-c, --concurrency INT processor threads to use
-d, --daemon Daemonize process
-e, --environment ENV Application environment
-g, --tag TAG Process tag for procline
-i, --index INT unique process index on this machine
-q, --queue QUEUE[,WEIGHT] Queues to process with optional weights
-r, --require [PATH|DIR] Location of Rails application with workers or file to require
-t, --timeout NUM Shutdown timeout
-v, --verbose Print more verbose output
-C, --config PATH path to YAML config file
-L, --logfile PATH path to writable logfile
-P, --pidfile PATH path to pidfile
-V, --version Print version and exit
-h, --help Show help

可以通过linux cli的方式,使用添数来启动sidekiq: bundle exec sidekiq -q queue_name_1,queue_name_2,也可以将这些参数放到yml中,通过 -C 参数来启动 bundle exec sidekiq -C config/sidekiq.yml

sidekiq 启动配置文件

config/sidekiq.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
:concurrency: 5
:pidfile: tmp/pids/sidekiq.pid
:queues:
- default
- [myqueue, 2]
development:
:concurrency: 5
staging:
:concurrency: 10
production:
:concurrency: 20

Worker

app/workers/sms_worker.rb
1
2
3
4
5
6
7
8
class SmsWorker
include Sidekiq::Worker
def perform(name, count)
# do something
p name
end
end

Run

配置好这些东西,就可以对刚才写的SmsWorker进行测试了。

首先,要启动Sidekiq:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ bundle exec sidekiq -C config/sidekiq.yml ✘
"localhost"
2014-06-10T05:08:07Z 36163 TID-ox8pyjylc INFO: Booting Sidekiq 3.1.3 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"highlander_sidekiq"}
s
ss
sss sss ss
s sss s ssss sss ____ _ _ _ _
s sssss ssss / ___|(_) __| | ___| | _(_) __ _
s sss \___ \| |/ _` |/ _ \ |/ / |/ _` |
s sssss s ___) | | (_| | __/ <| | (_| |
ss s s |____/|_|\__,_|\___|_|\_\_|\__, |
s s s |_|
s s
sss
sss

在另一个Term中使用Rails Console进行测试

1
2
3
4
$ rails c
# 调用perform
SmsWorker.perform_async 'Hello World', 0

就可以在sidekiq的调试窗口中看到输出了。

Deploy with Capistrano

添加gem capistrano-sidekiq添加到Rails Project

1
gem 'capistrano-sidekiq' , group: :development

然后,项目中运行’cap -vT’,可以看到添加下面这些有关sidekiq的命令。

1
2
3
4
5
6
cap sidekiq:quiet # Quiet sidekiq (stop processing new tasks)
cap sidekiq:respawn # Respawn missing sidekiq proccesses
cap sidekiq:restart # Restart sidekiq
cap sidekiq:rolling_restart # Rolling-restart sidekiq
cap sidekiq:start # Start sidekiq
cap sidekiq:stop # Stop sidekiq

(x)Cap 2.x中可以配置cmd来选择sidekiq.yml进行启动,但Cap 3.x中就只能过配在stage下纯代码的方式来配置(x),后来问了下作者,sidekiq_config就是用来在3.x中使用yml来启动的:

以下为cap-sidekiq的参数配置和其默认值:

1
2
3
4
5
6
7
8
9
10
11
12
13
:sidekiq_default_hooks => true
:sidekiq_pid => File.join(shared_path, 'tmp', 'pids', 'sidekiq.pid')
:sidekiq_env => fetch(:rack_env, fetch(:rails_env, fetch(:stage)))
:sidekiq_log => File.join(shared_path, 'log', 'sidekiq.log')
:sidekiq_options => nil
:sidekiq_require => nil
:sidekiq_tag => nil
:sidekiq_config => nil
:sidekiq_queue => nil
:sidekiq_timeout => 10
:sidekiq_role => :app
:sidekiq_processes => 1
:sidekiq_concurrency => nil

sidekiq_default_hooks 是capistrano-sidekiq提供的默认的hooks,在cap的指定task前后指行相应的task,如发布前关闭sidekiq,发布完后启动Sidekiq。具体见代码。如果不需要,可以设置该值为false,即可关闭。

Rails中的Cap配置示例:

示例如下,在stage中配置concurrency pool size和named queue的值

config/deploy/production.rb
1
2
3
4
5
6
## sidekiq
set :sidekiq_concurrency, 10
set :sidekiq_queue, ['client_sms,2', 'client_emails,5', 'default,3', 'foo']
## or use config.yml file
set :sidekiq_config, "#{current_path}/config/sidekiq.yml"

会启动如下命令:

1
RBENV_ROOT=~/.rbenv RBENV_VERSION=2.1.2 ~/.rbenv/bin/rbenv exec bundle exec sidekiq --index 0 --pidfile /home/ares/apps/highlander/shared/tmp/pids/sidekiq.pid --environment production --logfile /home/ares/apps/highlander/shared/log/sidekiq.log --queue sms, notification --concurrency 10 --daemon

这里有一个问题,我第一天使用时,花了半天去调试,发现是’set pry’的问题 Not started sidekiq after deploy - if pty is true

当初忘记为什么,在deploy.rb中设置pry为true了,只好在production stage的配置中,设置回来。

prod.rb
1
set :pty, false

进程监控

1
2
# 查看sidekiq进程
$ ps aux | grep sidekiq

使用自带的监控页面

Gemfile
1
gem 'sinatra'
route.rb
1
2
require 'sidekiq/web'
mount Sidekiq::Web => '/sidekiq'

这样,就可以在页面上看到sidekiq的运行情况了。更多详情见:sidekiq Monitoring

REF::

http://railscasts.com/episodes/366-sidekiq?view=asciicast
http://stackoverflow.com/questions/14825565/sidekiq-deploy-to-multiple-environments