S3 Object Storage REST APIs

Support for the S3 REST API

Object Storage supports the Simple Storage Service (S3) API, which is implemented as a set of Representational State Transfer (REST) web services. Support for the S3 REST API enables you to connect service-oriented applications developed for S3 web services with on-premises object storage that uses the Object Storage system. This requires minimal changes to a client application’s current use of S3 REST API calls.

Supported versions

Object Storage supports the following specific versions of S3 and HTTP.

ItemVersion
S3 specificationSimple Storage Service API Reference 2006-03-01
HTTP 1.1For more information about HTTP, see HTTP/1.1 (RFC 2616).

Support for Object Storage platform services

Object Storage platform services enable Object Storage tenant accounts to leverage external services such as a remote S3 bucket, a Simple Notification Service (SNS) endpoint, or an Elasticsearch cluster to extend the services provided by a grid. The following table summarizes the available platform services and the S3 APIs used to configure them.

Platform servicePurposeS3 API used to configure the service
CloudMirror replicationReplicates objects from a source Object Storage bucket to the configured remote S3 bucket.PUT Bucket replication
NotificationsSends notifications about events in a source Object Storage bucket to a configured Simple Notification Service (SNS) endpoint.PUT Bucket notification
Search integrationSends object metadata for objects stored in a Object Storage bucket to a configured Elasticsearch index.PUT Bucket metadata notification Note: This is a Object Storage custom S3 API.

A grid administrator must enable the use of platform services for a tenant account before they can be used. Then, a tenant administrator must create an endpoint that represents the remote service in the tenant account. This step is required before a service can be configured.

S3 REST API supported operations and limitations

The Object Storage system implements the Simple Storage Service API Reference (API Version 2006-03-01) with support for most operations, and with some limitations. You need to understand the implementation details when you are integrating S3 REST API client applications.

Date handling

The Object Storage implementation of the S3 REST API only supports valid HTTP date formats.

The Object Storage system only supports valid HTTP date formats for any headers that accept date values. The time portion of the date can be specified in Greenwich Mean Time (GMT) format, or in Universal Coordinated Time (UTC) format with no time zone offset (+0000 must be specified). If you include the x-amz-date header in your request, it overrides any value specified in the Date request header. When using AWS Signature Version 4, the x-amz-date header must be present in the signed request because the date header is not supported.

Support for Object Storage platform services

Object Storage platform services enable Object Storage tenant accounts to leverage external services such as a remote S3 bucket, a Simple Notification Service (SNS) endpoint, or an Elasticsearch cluster to extend the services provided by a grid.

The following table summarizes the available platform services and the S3 APIs used to configure them.

Platform servicePurposeS3 API used to configure the service
CloudMirror replicationReplicates objects from a source Object Storage bucket to the configured remote S3 bucket.PUT Bucket replication
NotificationsSends notifications about events in a source Object Storage bucket to a configured Simple Notification Service (SNS) endpoint.PUT Bucket notification
Search integrationSends object metadata for objects stored in a Object Storage bucket to a configured Elasticsearch index.PUT Bucket metadata notification Note: This is a Object Storage custom S3 API.

Testing your S3 REST API configuration

You can use the Amazon Web Services Command Line Interface (AWS CLI) to test your connection to the system and to verify that you can read and write objects to the system. Before you begin • You must have downloaded and installed the AWS CLI from aws.amazon.com/cli. • You must have created an S3 tenant account in the Object Storage system.

Steps

  1. Configure the Amazon Web Services settings to use the account you created in the Object Storage system:

  2. Enter configuration mode:
    aws configure --profile ProfileName

  3. Enter the AWS Access Key ID for the account you created.

  4. Enter the AWS Secret Access key for the account you created.

  5. Enter the default region to use, for example, us-east-1.

  6. Enter the default output format to use, for example, JSON.

  7. Create a bucket.

aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName  create-bucket --bucket testbucket
  • If the bucket is created successfully, the location of the bucket is returned, as seen in the following example:* "Location": "/testbucket"
  1. Upload an object.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName put-object --bucket testbucket --key s3.pdf --body C:\s3-test\upload\s3.pdf
  1. List the contents of the bucket to verify that the object was uploaded.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName list-objects --bucket testbucket
  1. Delete the object.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName delete-object --bucket testbucket --key s3.pdf
  1. Delete the bucket.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName delete-bucket --bucket testbucket

How Object Storage implements the S3 REST API

A client application can use S3 REST API calls to connect to Storage Nodes and API Gateway Nodes, to create buckets, and to store and retrieve objects.

Object versioning

You can use versioning to retain multiple versions of an object, which protects against accidental deletion of objects, and enables you to retrieve and restore earlier versions of an object.

The Object Storage system implements versioning with support for most features, and with some limitations. Object Storage supports up to 1,000 versions of each object.

In Object Storage, object versioning can be combined with Information Lifecycle Management (ILM), rather than Amazon’s Object Lifecycle Management, which is not supported. You must explicitly enable versioning for each bucket to turn on this functionality for the bucket. Each object in your bucket is assigned a version ID which is generated by the Object Storage system.

Common request headers

The Object Storage system supports common request headers defined by the Simple Storage Service API Reference, with several exceptions.

Request headerImplementation
Authorization•Full support for AWS Signature Version 2 •Support for AWS Signature Version 4, with the following exceptions: • The SHA256 value is not calculated for the body of the request. The user submitted value is accepted without validation.
x-amz-copy-source-server-side- encryption-customer-keyNot implemented.
x-amz-copy-source-server-side- encryption-customer-key-MD5Not implemented.
x-amz-security-tokenNot implemented. Returns XNotImplemented.
x-amz-server-side-encryption- customer-algorithmNot implemented.

Common response headers

The Object Storage system supports all of the common response headers defined by the Simple Storage Service API Reference, with several exceptions

Authenticating requests

The Object Storage system supports both authenticated and anonymous access to objects using the S3 API.

The S3 API supports Signature version 2 and Signature version 4 for authenticating S3 API requests.

Authenticated requests must be signed using your access key ID and secret access key.

The Object Storage system supports two authentication methods: the HTTP Authorization header and using query parameters.

Using the HTTP Authorization header The HTTP Authorization header is used by all S3 API operations except Anonymous requests where permitted by the bucket policy. The Authorization header contains all of the required signing information to authenticate a request.

Using query parameters You can use query parameters to add authentication information to a URL. This is known as presigning the URL, which can be used to grant temporary access to specific resources. Users with the presigned URL do not need to know the secret access key in order to access the resource, which enables you to provide third-party restricted access to a resource.

Operations on buckets

The Object Storage system supports a maximum of 1,000 buckets for each S3 tenant account.

Bucket name restrictions follow the AWS US Standard region restrictions, but you should further restrict them to DNS naming conventions in order to support S3 virtual hosted-style requests.

The GET Bucket (List Objects) and GET Bucket versions operations support Object Storage consistency controls.

You can check whether updates to last access time are enabled or disabled for individual buckets.

The following table describes how Object Storage implements S3 REST API bucket operations. To perform any of these operations, the necessary access credentials must be provided for the account.

OperationImplementation
DELETE BucketImplemented with all Amazon S3 REST API behavior.
DELETE Bucket corsThis operation deletes the CORS configuration for the bucket.
DELETE Bucket policyThis operation deletes the policy attached to the bucket.
DELETE Bucket replicationThis operation deletes the replication configuration attached to the bucket.
GET Bucket (List Objects), version 1This operation returns some or all (up to 1,000) of the objects in a bucket.
The Storage Class for objects is always listed as STANDARD, even if the object was ingested with the REDUCED_REDUNDANCY storage class option.
Requests that traverse a large number of keys require special handling. The request might return a truncated response or an empty response to avoid timing out.
To get the complete set of results, you need to continue making requests while updating the marker parameter, as you normally do with a truncated result. Always use NextMarker if it is present. In some cases, the Object Storage implementation of the S3 REST API returns a NextMarker, when the Amazon S3 REST API would not, because it is a better marker than the last key returned
GET Bucket aclThis operation returns a positive response and the ID, DisplayName, and Permission of the bucket owner, indicating that the owner has full access to the bucket.
GET Bucket corsThis operation returns the cors configuration for the bucket.
GET Bucket locationThis operation returns the bucket’s region. By default, us- east-1 is returned unless a region was set using the LocationConstraint element in the PUT Bucket request.
GET Bucket Object versionsWith READ access on a bucket, this operation with the versions subresource lists metadata of all of the versions of objects in the bucket.
GET Bucket notificationThis operation returns the notification configuration attached to the bucket.
GET Bucket policyThis operation returns the policy attached to the bucket.
GET Bucket replicationThis operation returns the replication configuration attached to the bucket.
GET Bucket versioningThis implementation uses the versioning subresource to return the versioning state of a bucket. The versioning state returned indicates if the bucket is “Unversioned” or if the bucket is version “Enabled” or “Suspended.”
HEAD BucketThis operation determines if a bucket exists and you have permission to access it.
PUT BucketThis operation creates a new bucket. By creating the bucket, you become the bucket owner.
• Bucket names must comply with the following rules:
o Must be unique across each Object Storage system (not just unique within the tenant account).
o Must be DNS compliant.
o Must contain between 3 and 63 characters.
o Can be a series of one or more labels, with adjacent labels separated by a period. Each label must start and end with a lowercase letter or a number and can only use lowercase letters, numbers, and hyphens.
o Must not look like a text-formatted IP address.
o Should not use periods in virtual hosted-style requests because periods will cause problems with server wildcard certificate verification.
• By
however, you can use the LocationConstraint request element in the request body to specify a different region. When using the LocationConstraint element, you must specify the exact name of a region that has been defined using the Grid Manager or the Grid Management API. Contact your system administrator if you do not know the region name you should use.
Note: An error will occur if your PUT Bucket request uses a region that has not been defined in Object Storage.
• When using a PUT Bucket request, you can include the SGCompliance XML element in the XML request body. SGCompliance is a custom Object Storage element that allows you to create buckets that are compliant.
default, buckets are created in the us-east-1 region;
PUT Bucket corsThis operation sets the CORS configuration for a bucket so that the bucket can service cross-origin requests.
Cross-origin resource sharing (CORS) is a security mechanism that allows client web applications in one domain to access resources in a different domain. For example, suppose you use an S3 bucket named images to store graphics. By setting the CORS configuration for the images bucket, you can allow the images in that bucket to be displayed on the website http:// www.example.com.
PUT Bucket notificationThis operation configures notifications for the bucket using the notification configuration XML included in the request body.
You should be aware of the following implementation details:
• Object Storage supports Simple Notification Service (SNS) topics as destinations. Simple Queue Service (SQS) or Amazon Lambda endpoints are not supported.
• The destination for notifications must be specified as the URN of an Object Storage endpoint. Endpoints can be created using the Tenant Manager or the Tenant Management API.
The endpoint must exist for notification configuration to succeed. If the endpoint does not exist, a 400 Bad Request error is returned with the code InvalidArgument.
• Notifications on the s3:ReducedRedundancyLostObject event are not supported.
• Event notification messages use standard values for most keys, except for the following:
o eventSource returns sgws:s3
o awsRegion: this key is not returned
o x-amz-id-2: this key is not returned
o arn returns urn:sgws:s3:::bucket-name
PUT Bucket policyThis operation sets the policy attached to the bucket.

Operations on objects

This section describes how the Object Storage system implements S3 REST API operations for objects.

The following conditions apply to all object operations:

• All of the operations on objects, except GET Object ACL and OPTIONS /, support Object Storage consistency controls.

• Conflicting client requests, such as two clients writing to the same key, are resolved on a “latest- wins” basis. The timing for the “latest-wins”evaluation is based on when the Object Storage system completes a given request, and not on when S3 clients begin an operation.

• All objects in a Object Storage bucket are owned by the bucket owner, including objects created by an anonymous user, or by another account.

• Data objects ingested to the Object Storage system through Swift cannot be accessed through S3.

OperationImplementation
DELETE ObjectMulti-Factor Authentication (MFA) and the response header x-amz- mfa are not supported.
Versioning
To remove a specific version, the requestor must be the bucket owner and use the versionId subresource. Using this subresource permanently deletes the version. If the versionId corresponds to a delete marker, the response header x-amz-delete-marker is returned set to true.
• If an object is deleted without the versionId subresource on a version enabled bucket, it results in the generation of a delete marker. The versionId for the delete marker is returned using the x-amz-version-id response header, and the x-amz-delete- marker response header is returned set to true.
• If an object is deleted without the versionId subresource on a version suspended bucket, it results in a permanent deletion of an already existing ‘null’ version or a ‘null’ delete marker, and the generation of a new ‘null’ delete marker. The x-amz-delete- marker response header is returned set to true.
DELETE Multiple ObjectsMulti-Factor Authentication (MFA) and the response header x-amz- mfa are not supported.
Multiple objects can be deleted in the same request message.
Note: The DELETE Multiple Objects request is not supported on versioned buckets.
Unlike the PUT Object operation, the DELETE Multiple Objects operation does not support chunked transfer encoding and the content encoding gzip attributes.
DELETE Object taggingUses the tagging subresource to remove all tags from an object. Implemented with all Amazon S3 REST API behavior.
Versioning
If the versionId query parameter is not specified in the request, the operation deletes all tags from the most recent version of the object in a versioned bucket.
GET ObjectThe following request headers are not supported and return
• x-amz-restore
• x-amz-website-redirect-location
Versioning
If a versionId subresource is not specified, the operation fetches the most recent version of the object in a versioned bucket. If the current version of the object is a delete marker, a “Not Found” status is returned with the x-amz-delete-marker response header set to true.
GET Object ACLIf the necessary access credentials are provided for the account, the operation returns a positive response and the ID, DisplayName, and Permission of the object owner, indicating that the owner has full access to the object.
GET Object taggingUses the tagging subresource to return all tags for an object. Implemented with all Amazon S3 REST API behavior
Versioning
If the versionId query parameter is not specified in the request, the operation returns all tags from the most recent version of the object in a versioned bucket.
HEAD ObjectThe following request headers are not supported and return XNotImplemented:
• x-amz-restore
• x-amz-website-redirect-location
Versioning
If a versionId subresource is not specified, the operation fetches the most recent version of the object in a versioned bucket. If the current version of the object is a delete marker, a “Not Found” status is returned with the x-amz-delete-marker response header set to true.
PUT Object taggingUses the tagging subresource to add a set of tags to an existing object. Implemented with all Amazon S3 REST API behavior Resolving conflicts
Conflicting client requests, such as two clients writing to the same key, are resolved on a “latest-wins” basis. The timing for the “latest- wins”evaluation is based on when the Object Storage system completes a given request, and not on when S3 clients begin an operation.
Versioning
If the versionId query parameter is not specified in the request, the operation add tags to the most recent version of the object in a versioned bucket. If the current version of the object is a delete marker, a “MethodNotAllowed” status is returned with the x-amz-delete- marker response header set to true.

Error responses

The Object Storage system supports all standard S3 REST API error responses that apply. In addition, the Object Storage implementation adds several custom responses.

Supported S3 API error codes

NameHTTP status
AccessDenied403 Forbidden
BadDigest400 Bad Request
BucketAlreadyExists409 Conflict
BucketNotEmpty409 Conflict
IncompleteBody400 Bad Request
InternalError500 Internal Server Error
InvalidAccessKeyId403 Forbidden
InvalidArgument400 Bad Request
InvalidBucketName400 Bad Request
InvalidDigest400 Bad Request
InvalidEncryptionAlgorithmError400 Bad Request
InvalidPart400 Bad Request
InvalidPartOrder400 Bad Request
InvalidRange416 Requested Range Not Satisfiable
InvalidRequest400 Bad Request
InvalidStorageClass400 Bad Request
InvalidTag400 Bad Request
InvalidURI400 Bad Request
KeyTooLong400 Bad Request
MalformedXML400 Bad Request
MetadataTooLarge400 Bad Request
MethodNotAllowed405 Method Not Allowed
MissingContentLength411 Length Required
MissingRequestBodyError400 Bad Request
MissingSecurityHeader400 Bad Request
NoSuchBucket404 Not Found
NoSuchKey404 Not Found
NoSuchUpload404 Not Found
NotImplemented501 Not Implemented
NoSuchBucketPolicy404 Not Found
PreconditionFailed412 Precondition Failed
RequestTimeTooSkewed403 Forbidden
ServiceUnavailable503 Service Unavailable
SignatureDoesNotMatch403 Forbidden
TooManyBuckets400 Bad Request
UserKeyMustBeSpecified400 Bad Request

Object Storage custom error codes |Name| Description| HTTP status | ——– | ——– | ——– | |XNoSuchBucketCompliance| The specified bucket does not have compliance enabled. | 404 Not Found | |XNotImplemented |The request you provided implies functionality that is not implemented. |501 Not Implemented |