Object Storage Service

Build Status Dependencies Cover

A microservice for object storage part of the Restorecommerce.

Features

  • Smart facade for S3 object storage exposed via gRPC.

  • It uses the AWS SDK client for connecting to S3 compatible stores.

  • Built in access control enforement based on the ACS.

  • Overcomes limitation of metadata in S3 by allowing to store meta data in redis.

Configuration

The following Object Storage Server configuration properties under the s3 configuration are available:

  • client.accessKeyId: access key ID for Object Storage Server.

  • client.secretAccessKey: secret access key for Object Storage Server.

  • client.endpoint: Object Storage Server endpoint.

  • client.s3ForcePathStyle: Whether to force path style URLs for S3 objects.

  • buckets: list of buckets to be created on server start up.

  • bucketsLifecycleConfigs.Bucket: name of bucket which receives the lifecycle configuration.

  • bucketsLifecycleConfigs.LifecycleConfiguration.Rules: array containing multiple predefined rules.

  • Rules.Status: status of predefined rule (e.g. Enabled | Disabled ).

  • Rules.Expiration.Date: the date when the expiration will occur. The value must conform to the ISO 8601 format. The time is always midnight UTC (e.g. '2019-05-30T00:00:00.000Z').

  • Rules.Expiration.Days: the number of days since object creation when the expiration will occur (e.g. 30). Amazon S3 calculates the time by adding the number of days specified in the rule to the object creation time and rounding the resulting time to the next day midnight UTC. For example, if an object was created at '1/15/2014 10:30 AM UTC' and you specify 3 days in a transition rule, then the transition date of the object would be calculated as '1/19/2014 00:00 UTC'.

  • Rules.Filter.Prefix: filtering based on prefix identifying one or more objects (files or folders) to which the rule applies (e.g. 'temp/' expires all objects under 'temp' folder).

  • Rules.Filter.Tag: filtering based on Tag which is identifying one or more objects to which the rule applies (e.g. 'Tag { Key: 'id_1', Value: 'value_1' }' expires all objects tagged with the respective key-value pair).

  • Rules.ID: unique identifier for the rule. The value cannot be longer than 255 characters.

  • authorization:enable: default value is true, if set to true then access-control check is performed, if value is set to false then access-control check is overridden.

  • authorization:enforce: default value is true, if set to true then result of access-control-check is enforced, if value is set to false the result of access-control check if logged and ignored.

  • redis:db-indexes:db-aclStore: Due to limitation of header size of object metadata in S3, meta.acl is stored in redis and this configuration is used to set the DB index in Redis.

API

This microservice exposes the following gRPC endpoints:

Put

Used to store the Object to the Storage Server. Requests are performed using io.restorecommerce.ostorage.Object protobuf message as input and response is io.restorecommerce.ostorage.Response message. For Object key characters described as safe to use in the S3 Object Key Naming Guidelines are allowed.

io.restorecommerce.ostorage.Object

Field Type Label Description

bucket

string

required

Bucket to which the object should be mapped to.

object

bytes

optional

Blob.

key

string

required

Object Key.

meta

io.restorecommerce.meta.Meta

optional

metadata attached to Object.

options

io.restorecommerce.ostorage.Options

optional

headers attached to Object.

subject

io.restorecommerce.auth.Subject

required

Subject details

io.restorecommerce.ostorage.Options

Field Type Label Description

encoding

string

optional

ContentEncoding header - Specifies what content encodings have been applied to the object and thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field

content_type

string

optional

ContentType header - A standard MIME type describing the format of the contents

content_language

string

optional

ContentLanguage header - The language the content is in

content_disposition

string

optional

ContentDisposition header - Specifies presentational information for the object

length

int32

optional

ContentLength header - Content size in bytes. This parameter is useful when the content size cannot be determined automatically

version

string

optional

x-amz-version-id header - Version ID of the newly created object, in case the bucket has versioning turned on

md5

string

optional

ETag - Entity tag that identifies the newly created object’s data

tags

[ ] io.restorecommerce.access_control.Attribute

optional

The tag-set for the object. The tag-set is used for filtering by tag

data

google.protobuf.Any

optional

additional data data provided as JSON object

io.restorecommerce.ostorage.Response

Field Type Label Description

url

string

required

Url of saved Object

bucket

string

required

Bucket to which the object is mapped to

key

string

optional

Object Key

meta

io.restorecommerce.meta.Meta

required

metadata attached to Object

tags

[ ] io.restorecommerce.access_control.Attribute

optional

The tag-set for the object. The tag-set is used for filtering by tag

length

int32

optional

Size of uploaded object

Get

Used to retrieve the Object from the Storage Server. Requests are performed using io.restorecommerce.ostorage.GetRequest protobuf message as input and response is io.restorecommerce.ostorage.Object message.

io.restorecommerce.ostorage.GetRequest

Field Type Label Description

key

string

required

Object Key

bucket

string

required

Bucket to which the object is mapped to

flag

boolean

optional

If flag is set to true only metadata of object is fetched

download

boolean

optional

If flag is set to true then Content-Disposition is set as attachment else is set as inline

subject

io.restorecommerce.auth.Subject

required

Subject details

List

Used to list all the Objects in a Bucket from the Storage Server. Requests are performed using io.restorecommerce.ostorage.ListRequest protobuf message as input and response is io.restorecommerce.ostorage.ListResponse message.

io.restorecommerce.ostorage.ListRequest

Field Type Label Description

bucket

string

optional

bucket name to list the Objects from, if bucket name is missing then all Objects from all buckets are listed

filters

io.restorecommerce.filter.FilterOp

optional

Filter based on fieldName, operation, value

max_keys

int32

optional

maximum number of keys to be returned in response

prefix

string

optional

prefix used list specific pattern key’s

subject

io.restorecommerce.auth.Subject

required

Subject details

io.restorecommerce.ostorage.ListResponse

Field Type Label Description

response

[ ] io.restorecommerce.ostorage.ObjectsDataWithPayloadStatus

required

List of Objects moved

operation_status

io.restorecommerce.status.OperationStatus

required

operation status

io.restorecommerce.ostorage.ObjectsDataWithPayloadStatus

Field Type Label Description

payload

io.restorecommerce.ostorage.ObjectData

required

Object payload

status

io.restorecommerce.status.Status

required

Object List status

io.restorecommerce.ostorage.ObjectData

Field Type Label Description

url

string

required

url for Object

object_name

string

required

Object name

meta

io.restorecommerce.meta.Meta

optional

metadata attached to Object

Move

Used to move Object safely from one bucket to another, can be used for renaming an Object as well. Requests are performed using io.restorecommerce.ostorage.MoveRequestList protobuf message as input and response is io.restorecommerce.ostorage.MoveResponseList message.

io.restorecommerce.ostorage.MoveRequestList

Field Type Label Description

items

[ ] io.restorecommerce.ostorage.MoveRequestItem

required

List of Objects to be moved

subject

io.restorecommerce.auth.Subject

required

Subject details

io.restorecommerce.ostorage.MoveRequestItem

Field Type Label Description

bucket

string

optional

destination bucket name to move the Object

key

string

optional

destination Object key name

sourceObject

string

optional

source bucket and key name

meta

io.restorecommerce.meta.Meta

optional

metadata attached to Object

options

io.restorecommerce.ostorage.Options

optional

headers attached to Object

io.restorecommerce.ostorage.MoveResponseList

Field Type Label Description

responses

[ ] io.restorecommerce.ostorage.MoveResponsePayloadWithStatus

required

List of Objects moved

operation_status

io.restorecommerce.status.OperationStatus

required

operation status

io.restorecommerce.ostorage.MoveResponsePayloadWithStatus

Field Type Label Description

payload

io.restorecommerce.ostorage.MoveResponseItem

required

Object payload

status

io.restorecommerce.status.Status

required

Object move status

io.restorecommerce.ostorage.MoveResponseItem

Field Type Label Description

bucket

string

required

destination bucket name

key

string

required

destination Object key name

sourceObject

string

required

source bucket and key name

meta

io.restorecommerce.meta.Meta

optional

metadata attached to Object

options

io.restorecommerce.ostorage.Options

optional

headers attached to Object

Delete

Used to delete the Object mapped to the Bucket from the Storage Server. Requests are performed using io.restorecommerce.ostorage.Bucket protobuf message as input and response is io.restorecommerce.resourcebase.DeleteResponse message.

io.restorecommerce.ostorage.DeleteRequest

Field Type Label Description

bucket

string

required

Bucket to which the object is mapped to

key

string

required

Object key

subject

io.restorecommerce.auth.Subject

required

Subject details

Copy

Used to copy the Object from one location to another in the Storage Server. If options are not provided, the object’s metadata is being preserved. When options are provided the metadata is being replaced.

Requests are performed using io.restorecommerce.ostorage.CopyRequestList protobuf message as input and response is io.restorecommerce.ostorage.CopyResponseList message.

The copy operation can also be used to update the metadata of an object (including tagging). In this case the concatenation of bucket and key input parameters should match the copySource string, in case if only object meta data information is specified in the io.restorecommerce.ostorage.CopyRequest and if tagging is not specified then by default the tags from source objects are copied to the new object.

io.restorecommerce.ostorage.CopyRequestList

Field Type Label Description

items

[] io.restorecommerce.ostorage.CopyRequestItem

required

List of objects to be copied

subject

io.restorecommerce.auth.Subject

required

Subject details

io.restorecommerce.ostorage.CopyRequestItem

Field Type Label Description

bucket

string

required

Destination bucket

copySource

string

required

Source path of object to be copied

key

string

required

Name of object to be created

meta

io.restorecommerce.meta.Meta

optional

user defined metadata attached to Object

options

io.restorecommerce.ostorage.Options

optional

object metadata attached to Object.

io.restorecommerce.ostorage.CopyResponseList

Field Type Label Description

responses

[ ] io.restorecommerce.ostorage.CopyResponsePayloadWithStatus

required

List of Objects copied

operation_status

io.restorecommerce.status.OperationStatus

required

operation status

io.restorecommerce.ostorage.CopyResponsePayloadWithStatus

Field Type Label Description

payload

io.restorecommerce.ostorage.CopyResponseItem

required

Copy Response payload

status

io.restorecommerce.status.Status

required

Object Copy status

io.restorecommerce.ostorage.CopyResponseItem

Field Type Label Description

bucket

string

required

Destination bucket

copySource

string

required

Source path of object to be copied

key

string

required

Name of object to be created

meta

io.restorecommerce.meta.Meta

optional

user defined metadata attached to Object

options

io.restorecommerce.ostorage.Options

optional

object metadata attached to Object.

io.restorecommerce.ostorage.OstorageMessage

OstorageMessage is used for firing events when objects are downloaded or uploaded:

Field Type Label Description

key

string

required

Object key

bucket

string

required

Bucket to which the object is mapped to

metadata

google.protobuf.Any

required

Object metadata

Events

Emitted

List of events emitted by this microservice for below topics:

Topic Name Event Name Description

io.restorecommerce.command

restoreResponse

system restore response

resetResponse

system reset response

healthCheckResponse

system health check response

versionResponse

system version response

configUpdateResponse

configuration update response

flushCacheResponse

flush ACS Cache response

io.restorecommerce.ostorage

objectUploaded

fired after an object was uploaded

objectDownloaded

fired after an object was downloaded

The objectUploaded and objectDownloaded events are using the message object io.restorecommerce.ostorage.OstorageMessage. Structure of message is defined above.

Consumed

This microservice consumes messages for the following events by topic:

Topic Name Event Name Description

io.restorecommerce.command

restoreCommand

for triggering for system restore

resetCommand

for triggering system reset

healthCheckCommand

to get system health check

versionCommand

to get system version

configUpdateCommand

used to update configurations

setApiKeyCommand

used to set API key

Note: currently restore and reset are not implemented.