Scheduling Service

A microservice for scheduling jobs part of the Restorecommerce.

Features

Scheduling of jobs in a cron like manner.
Three kinds of jobs can be scheduled: immediate jobs, one-time future jobs and recurring jobs.
Job scheduling is implemented using bull backed by Redis.
Concurrency and rate limiting for job processing. The job processing is made by matching the Job type with the Queue name. In the case when there is no match the jobs are being added to the default Queue.
Inbuilt mechanism to clean up completed / failed jobs.

Configuration

This service will hook any modules that are placed in the external-jobs directory. Use this to execute jobs on the scheduling-srv itself, the service looks for default export functions from each of the file in the external-jobs directory.
The rescheduling of missed recurring jobs on service start up can be disabled in the configuration.
Queues can be configured to use concurrency and rate limiting for job processing, where max is the maximum number of jobs processed, per duration in milliseconds.
Queues can be configured for advancedSettings for configuring lockDuration and other properties in the configuration, if the values are missing then bull configures to default values.
authorization:enable: default value is true, if set to true then access-control check is performed, if value is set to false then access-control check is overridden.
authorization:enforce: default value is true, if set to true then result of access-control-check is enforced, if value is set to false the result of access-control check if logged and ignored.
queueCleanup:cleanInterval: clean interval in milliseconds to remove completed / failed jobs from all queues.
queueCleanup:ttlAfterFinished: time to live after the job is in completed / failed state, default value is 604800000 (7 days) and this default configuration is applicable only if queueCleanup:cleanInterval is set.

API

This microservice exposes the following gRPC endpoints:

`Job`

io.restorecommerce.job.JobList

Field Type Label Description

Field	Type	Label	Description
items	`io.restorecommerce.job.Job`	required	List of Jobs.
total_count	number	optional	Number of Jobs
subject	`io.restorecommerce.auth.Subject`	required	Subject details

items

io.restorecommerce.job.Job

required

List of Jobs.

total_count

number

optional

Number of Jobs

subject

io.restorecommerce.auth.Subject

required

Subject details

io.restorecommerce.job.Job

Field	Type	Label	Description
id	string	optional	Job resource ID, if not specified an uuid is generated.
type	string	optional	Arbitrary job type (e.g: 'daily_email_dispatcher').
data	Data	optional	Job data to persist in Redis.
when	string	optional	Used to define the exact time at which a single job instance is processed. Ex: "Jan 15, 2018 10:30:00". This should only be used in one-time jobs.

Field

Type

Label

Description

string

optional

Job resource ID, if not specified an uuid is generated.

type

string

optional

Arbitrary job type (e.g: 'daily_email_dispatcher').

data

Data

optional

Job data to persist in Redis.

when

string

optional

Used to define the exact time at which a single job instance is processed. Ex: "Jan 15, 2018 10:30:00". This should only be used in one-time jobs.

io.restorecommerce.job.JobOptions

Field Type Label Description

Field	Type	Label	Description
priority	`io.restorecommerce.job.Job.Priority`	optional	Job priority.
attempts	number	optional	Amount of possible failing runs until a job fails.
backoff	`io.restorecommerce.job.Backoff`	optional	Delay settings between failed job runs.
timeout	number	optional	The number of milliseconds after which the job should be fail with a timeout error.
repeat	`io.restorecommerce.job.Repeat`	optional	Job repeat options

priority

io.restorecommerce.job.Job.Priority

optional

Job priority.

attempts

number

optional

Amount of possible failing runs until a job fails.

backoff

io.restorecommerce.job.Backoff

optional

Delay settings between failed job runs.

timeout

number

optional

The number of milliseconds after which the job should be fail with a timeout error.

repeat

io.restorecommerce.job.Repeat

optional

Job repeat options

io.restorecommerce.job.Repeat

Field	Type	Label	Description
every	number	optional	Interval to run a job periodically in milliseconds (cron setting cannot be used together with this setting).
cron	string	optional	Interval to run a job periodically in a cron format (e.g: "0 0 5 * * *"). This should only be used in recurring jobs.
startDate	string	optional	Used to define the exact time at which job should start repeating (only with cron). Ex: "Jan 15, 2018 10:30:00".
endDate	string	optional	Used to define the exact time at which job should stop repeating Ex: "Jan 15, 2018 10:30:00".
count	number	optional	How many times a job has repeated.

Field

Type

Label

Description

every

number

optional

Interval to run a job periodically in milliseconds (cron setting cannot be used together with this setting).

cron

string

optional

Interval to run a job periodically in a cron format (e.g: "0 0 5 * * *"). This should only be used in recurring jobs.

startDate

string

optional

Used to define the exact time at which job should start repeating (only with cron). Ex: "Jan 15, 2018 10:30:00".

endDate

string

optional

Used to define the exact time at which job should stop repeating Ex: "Jan 15, 2018 10:30:00".

count

number

optional

How many times a job has repeated.

io.restorecommerce.job.Data

Field Type Label Description

Field	Type	Label	Description
timezone	string	optional	Timezone specification for job scheduling (ex: 'Europe/Amsterdam').
meta	`io.restorecommerce.meta.Meta`	required	Job resource meta info; only contains creation and modification timestamps.
payload	[ ] `google.protobuf.Any`	optional	Generic data type for job-specific data.

timezone

string

optional

Timezone specification for job scheduling (ex: 'Europe/Amsterdam').

Name	Number	Description
NORMAL	0	normal priority, default value
LOW	10	low priority
MEDIUM	-5	medium priority
HIGH	-10	high priority
CRITICAL	-15	critical priority

Field	Type	Label	Description
delay	number	optional	Time until retry in milliseconds.
type	`io.restorecommerce.job.Backoff.Type`	optional	Calculation of the delay for retries.

Name	Number	Description
FIXED	0	Retry with the same delay.
EXPONENTIAL	1	Exponential delay increase between retries.

CRUD Operations

The microservice exposes the below CRUD operations for creating or modifying Job resources.

io.restorecommerce.job.JobService

Method Name Request Type Response Type Description

Method Name	Request Type	Response Type	Description
Create	`io.restorecommerce.job.JobList`	`io.restorecommerce.job.JobList`	Create a list of Job resources.
Read	`io.restorecommerce.job.JobReadRequest`	`io.restorecommerce.job.JobList`	Read a list of Job resources.
Update	`io.restorecommerce.job.JobList`	`io.restorecommerce.job.JobList`	Update a list of Job resources.
Delete	`io.restorecommerce.resourcebase.DeleteRequest`	`google.protobuf.Empty`	Delete a list of Job resources.

Create

io.restorecommerce.job.JobList

Create a list of Job resources.

Read

io.restorecommerce.job.JobReadRequest

io.restorecommerce.job.JobList

Read a list of Job resources.

Update

io.restorecommerce.job.JobList

Update a list of Job resources.

Delete

io.restorecommerce.resourcebase.DeleteRequest

google.protobuf.Empty

Delete a list of Job resources.

`Create`

Used to create list of Job resources. Requests are performed providing io.restorecommerce.job.JobList protobuf message as input and responses are io.restorecommerce.job.JobList message. If job id is provided then it is used for job identifier (should be unique), if not then an uuid is generated from this service. For repeatable Jobs Bull has an internal mechanism to generate job id, it generates the repeatable key based on the job id and cron settings. To map the repeatable key to job identifier exposed from this service Redis is used with configurable DB index to store the key value pair.

`Read`

Used to read list of Job resources. Requests are performed providing io.restorecommerce.job.JobReadRequest protobuf message as input and responses are io.restorecommerce.job.JobList message. This operation supports job filter using job ids or job types. If no job filter is provided then all jobs in redis with job state completed, failed, active, inactive jobs in redis are returned. For the detailed protobuf message structure of io.restorecommerce.job.ReadRequest refer job.proto.

io.restorecommerce.job.JobReadRequest

Field Type Label Description

Field	Type	Label	Description
sort	`io.restorecommerce.job.SortOrder`	optional	sort order
filter	`io.restorecommerce.job.JobFilter`	optional	job filter
fields	`io.restorecommerce.resourcebase.FieldFilter`	field filter	subject

sort

io.restorecommerce.job.SortOrder

optional

sort order

filter

io.restorecommerce.job.JobFilter

optional

job filter

fields

io.restorecommerce.resourcebase.FieldFilter

field filter

subject

io.restorecommerce.job.SortOrder

Field	Type	Label	Description
UNSORTED	ENUM	optional	unsorted order
ASCENDING	ENUM	optional	ascending order
DESCENDING	ENUM	optional	descending order

Field

Type

Label

Description

UNSORTED

ENUM

optional

unsorted order

ASCENDING

ENUM

optional

ascending order

DESCENDING

ENUM

optional

descending order

io.restorecommerce.job.JobFilter

Field	Type	Label	Description
job_ids	string []	optional	list of job identifiers
type	string	optional	job type

Field

Type

Label

Description

job_ids

string []

optional

list of job identifiers

type

string

optional

job type

`Update`

Used to update list of Job resources. Requests are performed providing io.restorecommerce.job.JobList protobuf message as input and responses are io.restorecommerce.job.JobList message. Since Bull does not internally provide an update operation, this operation deletes an existing job and reschedules it with new data. It is possible to update the cron settings and other job options as well for recurring job based on the job-id. In this case although the repeatable key maintained by Bull is changed the job id exposed from this service remains the same (since this service maintains a mapping between job id and repeatable key).

`Upsert`

Used to upsert list of Job resources. Requests are performed providing io.restorecommerce.job.JobList protobuf message as input and responses are io.restorecommerce.job.JobList message. If job does not exist then a new job is created. If job already exists with given job id, then it is deleted, and a new job is created with provided data, but the job identifier remains the same.

`Delete`

Used to delete list of Job resources. Requests are performed providing io.restorecommerce.resourcebase.DeleteRequest protobuf message as input and responses are a google.protobuf.Empty message. This operation supports passing a list of job identifiers ids to delete, it also exposes boolean param collection and if set to true all the jobs in Redis data would be deleted. For the detailed protobuf message structure of`io.restorecommerce.job.DeleteRequest` refer job.proto.

there is a folder external-jobs in the repository where any custom job processors can be placed.

Events

Emitted

List of events emitted by this microservice for below topics:

Topic Name Event Name Description

Topic Name	Event Name	Description
io.restorecommerce.jobs	jobsCreated	emitted when a job is created.
	jobsDeleted	emitted when a job is deleted.
io.restorecommerce.command	restoreResponse	system restore response.
	resetResponse	system reset response.
	healthCheckResponse	system health check response.
	versionResponse	system version response.
	`flushCacheResponse`	flush ACS Cache response

io.restorecommerce.jobs

jobsCreated

emitted when a job is created.

jobsDeleted

emitted when a job is deleted.

io.restorecommerce.command

restoreResponse

system restore response.

resetResponse

system reset response.

healthCheckResponse

system health check response.

versionResponse

system version response.

flushCacheResponse

flush ACS Cache response

Jobs can be created, updated or deleted by issuing Kafka messages to topic io.restorecommerce.jobs. These operations are exposed with the same input as the gRPC endpoints (note that it is only possible to read a job through gRPC).

io.restorecommerce.job.ScheduledJob

Field Type Label Description

Field	Type	Label	Description
id	number	optional	Job instance ID in Redis.
type	string	optional	Arbitrary job type (e.g: 'daily_email_dispatcher').
data	`io.restorecommerce.job.Data`	optional	Arbitrary job type (e.g: 'daily_email_dispatcher').
schedule_type	string	optional	Job type ex: `ONCE`, `RECURR` etc.

number

optional

Job instance ID in Redis.

type

string

optional

Arbitrary job type (e.g: 'daily_email_dispatcher').

data

io.restorecommerce.job.Data

optional

Arbitrary job type (e.g: 'daily_email_dispatcher').

schedule_type

string

optional

Job type ex: ONCE, RECURR etc.

io.restorecommerce.job.JobDone

Field Type Label Description

Field	Type	Label	Description
id	number	optional	Job instance ID in Redis.
schedule_type	string	optional	Job type ex: `ONCE`, `RECURR` etc.
delete_scheduled	boolean	optional	Whether to delete this repeating job.
type	string	optional	Job Type (required if emitting back a message).
result	`google.protobuf.Any`	optional	Generic data type for job-specific data.

number

optional

Job instance ID in Redis.

schedule_type

string

optional

Job type ex: ONCE, RECURR etc.

delete_scheduled

boolean

optional

Whether to delete this repeating job.

type

string

optional

Job Type (required if emitting back a message).

result

google.protobuf.Any

optional

Generic data type for job-specific data.

io.restorecommerce.job.JobFailed

Field Type Label Description

Field	Type	Label	Description
id	number	optional	Job instance ID in redis.
schedule_type	string	optional	Job type ex: `ONCE`, `RECURR` etc.
error	string	optional	Failure details.

number

optional

Job instance ID in redis.

schedule_type

string

optional

Job type ex: ONCE, RECURR etc.

error

string

optional

Failure details.

Events from the io.restorecommerce.jobs topic are issued whenever a CRUD operation is performed. They are useful for rescheduling of jobs in case of Redis failure.

Jobs emitted by this service to Kafka can be consumed by other microservices by listening to the queuedJob event. After processing the job an event should be emitted by the respective microservice indicating job failure or completion. A job is always deleted upon being receiving failure or completion data, unless it is a recurring job.

Consumed

This microservice consumes messages for the following events by topic:

Topic Name Event Name Description

Topic Name	Event Name	Description
io.restorecommerce.jobs	createJobs	for creating jobs
	modifyJobs	for modifying specific jobs
	deleteJobs	for deleting jobs
	jobDone	for when a job has finished
	jobFailed	for when a job has failed
io.restorecommerce.command	restoreCommand	for triggering for system restore
	resetCommand	for triggering system reset
	healthCheckCommand	to get system health check
	versionCommand	to get system version
	`flushCacheCommand`	used to flush ACS Cache
	`configUpdateCommand`	used to update configurations

io.restorecommerce.jobs

createJobs

for creating jobs

modifyJobs

for modifying specific jobs

deleteJobs

for deleting jobs

jobDone

for when a job has finished

jobFailed

for when a job has failed

io.restorecommerce.command

restoreCommand

for triggering for system restore

resetCommand

for triggering system reset

healthCheckCommand

to get system health check

versionCommand

to get system version

flushCacheCommand

used to flush ACS Cache

configUpdateCommand

used to update configurations