Background jobs with php and resque: part 5, creating jobs

Now that you have some workers running, let’s feed them some jobs. In this part, we’ll try to understand what’s a job, and how to refactor your application to implement background jobs.

What’s a job A job is a written order to tell the workers to execute a particular task. This order looks like:

Mail, dest@mail.com, hi!, "this is a test content". Order can only contain string. It’s that string that’ll be pushed in the queue (enqueued).

Queuing a job First of all, your application must see the

php-resque library. Let’s require it:
require_once ‘/path/to/php-resque/lib/Resque.php’;
It must also be able to connect to your

Redis server. If your server isn’t located at the default localhost:6379, your must tell resque where to find it:
Resque::setBackend(‘192.168.1.56:6380’); # For example
Your application is now ready, let’s enqueue a job.

Resque::enqueue('default', 'Mail',  array('dest@mail.com', 'hi!', 'this is a test content'));
  • First argument is the queue name. In that example, the job is pushed in the default queue.
  • Second argument is the job classname.
  • Third argument is the job’s arguments. You could also use an associative array like:
Resque::enqueue('default', 'Mail', array(
    'to' => 'dest@mail.com', 
    'subject' => 'hi!', 
    'body' => 'this is a test content'
);
Third argument can also be a simple string, but more data can be passed if using an array. Array can only contain string, or array of string. You can’t pass object like $this.

All job’s arguments are

json_encode() before pushed in the queue.

Executing a job Considering that you have created a worker to poll the

default queue, the worker will then pull the order from the queue, and parse it.
* The first token of Resque::enqueue() is the queue name. Worker doesn’t need it.
* The second token (Mail) is the class to instantiate
* The third: an array of arguments for the job The

Mail class is called a job class.

The Job class All job classes must implement a

perform() method. The third argument passed to Resque::enqueue() will be available inside the job class via `$this->args’. A typical job class looks like:

class Mail
{
    public function perform() {
        # With the first enqueue() example
        # $this->args = array('dest@mail.com', 'hi!', 'this is a test content');

        # With the second enqueue() example
        # $this->args = array(
        #    'to' => 'dest@mail.com', 
        #    'subject' => 'hi!', 
        #    'body' => 'this is a test content'
        # );
    }
}

Job can also have a

setUp and tearDown method. If defined, setUp will be run before perform, and tearDown after perform.

class Mail
{
    public function setUp() {
       # Set up something before perform, like establishing a database connection
    }

    public function perform() {
       # Do your job
    }

    public function tearDown() {
       # Run after perform, like closing resources
    }
}

Before instantiating that job class, the worker must find it first.

Locating the job All your jobs classes must be discoverable by the workers. There’s more than one way to do that.

Using include_path You can put all your job classes in php include path.

Via .htaccess Assuming that your PHP is running as an Apache module

php_value include_path ".:/already/existing/path:/path/to/job-classes"

Via php.ini

include_path = ".:/php/includes:/path/to/job-classes"
You can’t use the php version set_include_path(), as this only last for the duration of the script.

Using APP_INCLUDE Remember that

APP_INCLUDE parameter when creating workers ? It’s the path to a file that will be included by the worker.
QUEUE=default APP_INCLUDE=/path/file.php php resque
There are also multiple way to use that file:

With set_include_path() There, you

can use set_include_path() since we’re now inside the worker’s “scope”.

set_include_path(get_include_path() . PATH_SEPARATOR . "/path/to/jobs");

Not so useful when your job classes are scattered everywhere.

With require or include Require/Include all your job classes individually

include '/path/to/Mail.php';
include '/path/to/AnotherJobClass.php';
include '/path/to/somewhere/AnotherJobClass.php';
include '/JobClass.php';

A hell to manage when you have a lot of classes.

With an autoloader Use php

spl_autoloader. Chances are high that your application already have one. This is the recommend method.

Refactoring your code There exist 2 ways to execute some bit of your application code as background job.

Move the code to a job class Transform that

class User 
{
    # various functions(){}

    public function updateLocation($location) {
        $db->updateUserTable($this->userId, 'location', $location);
        $this->recomputeNewFriends(); # <- execute as background job
    }

    public function recomputeNewFriends() {
        # search for new possible friends
    }
}

into this

class User 
{
    # various function(){}

    public function updateLocation($location) {
        $db->updateUserTable($this->userId, 'location', $location);
        # Enqueue the job     
        Resque::enqueue('queueName', 'FriendRecommendator', array('id' => $this->userId));
    }
}

And this is the corresponding job class

class FriendRecommendator 
{
    function perform() {
        # We don't have access to $user object anymore, create it if needed
        $user = new User($this->args['id']);
        # search for new possible friends
    }
}

The User class will not have direct access to the new friends computation code anymore. If you still want it, you have to do that:

public function updateLocation($location) {
    $db->updateUserTable($this->userId, 'location', $location);
    $friendRecommendator = new FriendRecommendator();
    $friendRecommendator->args = array('id' => $this->userId);
    $friendRecommendator->perform();
}

Make your regular classes implements the Job interface Instead of moving the task to a job class, you can “convert” your regular class to a job class. Just add a

perform() method. Use in last resort, this is not recommended, this is just a possibility.

Drawbacks

  • The class implement a perform() method
  • Worker will manipulate the class’ $args property
  • The class can only have one background job The last point is also true for the standalone job class method. We can hack the job class a bit to make them support executing more than one job.

Group your jobs by class At this point, you have one class per job. This is the default structure defined by Resque.

Job class structure What follow is a hacking tip to group jobs belonging to the same family into a single class. Let’s consider the following job classes:

# SendNotificationToFriend.php
class SendNotificationToFriend
{
    public function perform() {}
}

# SendNotificationToAdmin.php
class SendNotificationToFriend
{
    public function perform() {}
}

You’ll want to group them in a Notification class:

# Notification.php
class Notification
{
    public function sendToFriend() {}

    public function sendToAdmin() {}

    public function perform() {}
}

This is particulary useful when the functions share the

setUp and tearDown method. To make this works, we have to add the function name to execute when enqueuing the job.
Resque::enqueue(‘queueName’, ‘Notification’, array(‘sendToFriends’, ‘otherArgs’);
We use the first element of the third argument to store the function name. Let’s edit the

perform() method to execute the right function now:

public function perform() {
    $this->{array_shift($this->args)}();
}

perform() is now the same for all job classes, let’s use some inheritance:

# Job.php
class Job
{
    public function perform() {
        $this->{array_shift($this->args)}();
    }
}

Then make all your job classes extends this Job Class:

# Notification.php
class Notification extends Job
{
    public function sendToFriend() {}

    public function sendToAdmin() {)
}

Et voila ! Neat job classes that you can re-use anywhere.

Remember The third argument in Resque::enqueue() is an array. This array first element must always be the job method name you want to execute. This element is removed from $args once inside the method.
Restart your workers each time you edit your job classes.

Track your jobs All workers activities are

logged. You’ll find in the logs:
* When a job is enqueued with Resque::enqueue
* When a worker detects a new job in the queue
* When a worker begins executing a job
* The status of the job execution (success/fail) Depending on your verbose mode, they come with more or less details. You can also use the redis monitor command to monitor redis activities.

redis-cli monitor

Next time At this point, you know how to use background jobs in a generic php application. Next parts of this tutorial serie will focus on third-party tools to get the most out of Resque. In the

next part (6), we’ll see how to implement background jobs in the cakePHP framework.