Scheduling Service

Build Status Dependencies Coverage Status

A microservice for scheduling jobs part of the Restorecommerce.

Features

  • Scheduling of jobs in a cron like manner.

  • Three kinds of jobs can be scheduled: immediate jobs, one-time future jobs and recurring jobs.

  • Job scheduling is implemented using bull backed by Redis.

  • Concurrency and rate limiting for job processing. The job processing is made by matching the Job type with the Queue name. In the case when there is no match the jobs are being added to the default Queue.

  • Inbuilt mechanism to clean up completed / failed jobs.

Configuration

  • This service will hook any modules that are placed in the external-jobs directory. Use this to execute jobs on the scheduling-srv itself, the service looks for default export functions from each of the file in the external-jobs directory.

  • The rescheduling of missed recurring jobs on service start up can be disabled in the configuration.

  • Queues can be configured to use concurrency and rate limiting for job processing, where max is the maximum number of jobs processed, per duration in milliseconds.

  • Queues can be configured for advancedSettings for configuring lockDuration and other properties in the configuration, if the values are missing then bull configures to default values.

  • authorization:enable: default value is true, if set to true then access-control check is performed, if value is set to false then access-control check is overridden.

  • authorization:enforce: default value is true, if set to true then result of access-control-check is enforced, if value is set to false the result of access-control check if logged and ignored.

  • queueCleanup:cleanInterval: clean interval in milliseconds to remove completed / failed jobs from all queues.

  • queueCleanup:ttlAfterFinished: time to live after the job is in completed / failed state, default value is 604800000 (7 days) and this default configuration is applicable only if queueCleanup:cleanInterval is set.

API

This microservice exposes the following gRPC endpoints:

Job

io.restorecommerce.job.JobList

Field Type Label Description

items

io.restorecommerce.job.Job

required

List of Jobs.

total_count

number

optional

Number of Jobs

subject

io.restorecommerce.auth.Subject

required

Subject details

io.restorecommerce.job.Job

Field Type Label Description

id

string

optional

Job resource ID, if not specified an uuid is generated.

type

string

optional

Arbitrary job type (e.g: 'daily_email_dispatcher').

data

Data

optional

Job data to persist in Redis.

when

string

optional

Used to define the exact time at which a single job instance is processed. Ex: "Jan 15, 2018 10:30:00". This should only be used in one-time jobs.

io.restorecommerce.job.JobOptions

Field Type Label Description

priority

io.restorecommerce.job.Job.Priority

optional

Job priority.

attempts

number

optional

Amount of possible failing runs until a job fails.

backoff

io.restorecommerce.job.Backoff

optional

Delay settings between failed job runs.

timeout

number

optional

The number of milliseconds after which the job should be fail with a timeout error.

repeat

io.restorecommerce.job.Repeat

optional

Job repeat options

io.restorecommerce.job.Repeat

Field Type Label Description

every

number

optional

Interval to run a job periodically in milliseconds (cron setting cannot be used together with this setting).

cron

string

optional

Interval to run a job periodically in a cron format (e.g: "0 0 5 * * *"). This should only be used in recurring jobs.

startDate

string

optional

Used to define the exact time at which job should start repeating (only with cron). Ex: "Jan 15, 2018 10:30:00".

endDate

string

optional

Used to define the exact time at which job should stop repeating Ex: "Jan 15, 2018 10:30:00".

count

number

optional

How many times a job has repeated.

io.restorecommerce.job.Data

Field Type Label Description

timezone

string

optional

Timezone specification for job scheduling (ex: 'Europe/Amsterdam').

meta

io.restorecommerce.meta.Meta

required

Job resource meta info; only contains creation and modification timestamps.

payload

[ ] google.protobuf.Any

optional

Generic data type for job-specific data.

io.restorecommerce.job.Job.Priority

Name Number Description

NORMAL

0

normal priority, default value

LOW

10

low priority

MEDIUM

-5

medium priority

HIGH

-10

high priority

CRITICAL

-15

critical priority

io.restorecommerce.job.Backoff

Field Type Label Description

delay

number

optional

Time until retry in milliseconds.

type

io.restorecommerce.job.Backoff.Type

optional

Calculation of the delay for retries.

io.restorecommerce.job.Backoff.Type

Name Number Description

FIXED

0

Retry with the same delay.

EXPONENTIAL

1

Exponential delay increase between retries.

CRUD Operations

The microservice exposes the below CRUD operations for creating or modifying Job resources.

io.restorecommerce.job.JobService

Method Name Request Type Response Type Description

Create

io.restorecommerce.job.JobList

io.restorecommerce.job.JobList

Create a list of Job resources.

Read

io.restorecommerce.job.JobReadRequest

io.restorecommerce.job.JobList

Read a list of Job resources.

Update

io.restorecommerce.job.JobList

io.restorecommerce.job.JobList

Update a list of Job resources.

Delete

io.restorecommerce.resourcebase.DeleteRequest

google.protobuf.Empty

Delete a list of Job resources.

Create

Used to create list of Job resources. Requests are performed providing io.restorecommerce.job.JobList protobuf message as input and responses are io.restorecommerce.job.JobList message. If job id is provided then it is used for job identifier (should be unique), if not then an uuid is generated from this service. For repeatable Jobs Bull has an internal mechanism to generate job id, it generates the repeatable key based on the job id and cron settings. To map the repeatable key to job identifier exposed from this service Redis is used with configurable DB index to store the key value pair.

Read

Used to read list of Job resources. Requests are performed providing io.restorecommerce.job.JobReadRequest protobuf message as input and responses are io.restorecommerce.job.JobList message. This operation supports job filter using job ids or job types. If no job filter is provided then all jobs in redis with job state completed, failed, active, inactive jobs in redis are returned. For the detailed protobuf message structure of io.restorecommerce.job.ReadRequest refer job.proto.

io.restorecommerce.job.JobReadRequest

Field Type Label Description

sort

io.restorecommerce.job.SortOrder

optional

sort order

filter

io.restorecommerce.job.JobFilter

optional

job filter

fields

io.restorecommerce.resourcebase.FieldFilter

field filter

subject

io.restorecommerce.job.SortOrder

Field Type Label Description

UNSORTED

ENUM

optional

unsorted order

ASCENDING

ENUM

optional

ascending order

DESCENDING

ENUM

optional

descending order

io.restorecommerce.job.JobFilter

Field Type Label Description

job_ids

string []

optional

list of job identifiers

type

string

optional

job type

Update

Used to update list of Job resources. Requests are performed providing io.restorecommerce.job.JobList protobuf message as input and responses are io.restorecommerce.job.JobList message. Since Bull does not internally provide an update operation, this operation deletes an existing job and reschedules it with new data. It is possible to update the cron settings and other job options as well for recurring job based on the job-id. In this case although the repeatable key maintained by Bull is changed the job id exposed from this service remains the same (since this service maintains a mapping between job id and repeatable key).

Upsert

Used to upsert list of Job resources. Requests are performed providing io.restorecommerce.job.JobList protobuf message as input and responses are io.restorecommerce.job.JobList message. If job does not exist then a new job is created. If job already exists with given job id, then it is deleted, and a new job is created with provided data, but the job identifier remains the same.

Delete

Used to delete list of Job resources. Requests are performed providing io.restorecommerce.resourcebase.DeleteRequest protobuf message as input and responses are a google.protobuf.Empty message. This operation supports passing a list of job identifiers ids to delete, it also exposes boolean param collection and if set to true all the jobs in Redis data would be deleted. For the detailed protobuf message structure of`io.restorecommerce.job.DeleteRequest` refer job.proto.

there is a folder external-jobs in the repository where any custom job processors can be placed.

Events

Emitted

List of events emitted by this microservice for below topics:

Topic Name Event Name Description

io.restorecommerce.jobs

jobsCreated

emitted when a job is created.

jobsDeleted

emitted when a job is deleted.

io.restorecommerce.command

restoreResponse

system restore response.

resetResponse

system reset response.

healthCheckResponse

system health check response.

versionResponse

system version response.

flushCacheResponse

flush ACS Cache response

setApiKeyResponse

set API Key response

Jobs can be created, updated or deleted by issuing Kafka messages to topic io.restorecommerce.jobs. These operations are exposed with the same input as the gRPC endpoints (note that it is only possible to read a job through gRPC).

io.restorecommerce.job.ScheduledJob

Field Type Label Description

id

number

optional

Job instance ID in Redis.

type

string

optional

Arbitrary job type (e.g: 'daily_email_dispatcher').

data

io.restorecommerce.job.Data

optional

Arbitrary job type (e.g: 'daily_email_dispatcher').

schedule_type

string

optional

Job type ex: ONCE, RECURR etc.

io.restorecommerce.job.JobDone

Field Type Label Description

id

number

optional

Job instance ID in Redis.

schedule_type

string

optional

Job type ex: ONCE, RECURR etc.

delete_scheduled

boolean

optional

Whether to delete this repeating job.

type

string

optional

Job Type (required if emitting back a message).

result

google.protobuf.Any

optional

Generic data type for job-specific data.

io.restorecommerce.job.JobFailed

Field Type Label Description

id

number

optional

Job instance ID in redis.

schedule_type

string

optional

Job type ex: ONCE, RECURR etc.

error

string

optional

Failure details.

Events from the io.restorecommerce.jobs topic are issued whenever a CRUD operation is performed. They are useful for rescheduling of jobs in case of Redis failure.

Jobs emitted by this service to Kafka can be consumed by other microservices by listening to the queuedJob event. After processing the job an event should be emitted by the respective microservice indicating job failure or completion. A job is always deleted upon being receiving failure or completion data, unless it is a recurring job.

Consumed

This microservice consumes messages for the following events by topic:

Topic Name Event Name Description

io.restorecommerce.jobs

createJobs

for creating jobs

modifyJobs

for modifying specific jobs

deleteJobs

for deleting jobs

jobDone

for when a job has finished

jobFailed

for when a job has failed

io.restorecommerce.command

restoreCommand

for triggering for system restore

resetCommand

for triggering system reset

healthCheckCommand

to get system health check

versionCommand

to get system version

flushCacheCommand

used to flush ACS Cache

configUpdateCommand

used to update configurations