Support for the S3 REST API
Object Storage supports the Simple Storage Service (S3) API, which is implemented as a set of Representational State Transfer (REST) web services. Support for the S3 REST API enables you to connect service-oriented applications developed for S3 web services with on-premises object storage that uses the Object Storage system. This requires minimal changes to a client application’s current use of S3 REST API calls.
Supported versions
Object Storage supports the following specific versions of S3 and HTTP.
| Item | Version |
|---|---|
| S3 specification | Simple Storage Service API Reference 2006-03-01 |
| HTTP 1.1 | For more information about HTTP, see HTTP/1.1 (RFC 2616). |
Support for Object Storage platform services
Object Storage platform services enable Object Storage tenant accounts to leverage external services such as a remote S3 bucket, a Simple Notification Service (SNS) endpoint, or an Elasticsearch cluster to extend the services provided by a grid. The following table summarizes the available platform services and the S3 APIs used to configure them.
| Platform service | Purpose | S3 API used to configure the service |
|---|---|---|
| CloudMirror replication | Replicates objects from a source Object Storage bucket to the configured remote S3 bucket. | PUT Bucket replication |
| Notifications | Sends notifications about events in a source Object Storage bucket to a configured Simple Notification Service (SNS) endpoint. | PUT Bucket notification |
| Search integration | Sends object metadata for objects stored in a Object Storage bucket to a configured Elasticsearch index. | PUT Bucket metadata notification Note: This is a Object Storage custom S3 API. |
A grid administrator must enable the use of platform services for a tenant account before they can be used. Then, a tenant administrator must create an endpoint that represents the remote service in the tenant account. This step is required before a service can be configured.
S3 REST API supported operations and limitations
The Object Storage system implements the Simple Storage Service API Reference (API Version 2006-03-01) with support for most operations, and with some limitations. You need to understand the implementation details when you are integrating S3 REST API client applications.
Date handling
The Object Storage implementation of the S3 REST API only supports valid HTTP date formats.
The Object Storage system only supports valid HTTP date formats for any headers that accept date values. The time portion of the date can be specified in Greenwich Mean Time (GMT) format, or in Universal Coordinated Time (UTC) format with no time zone offset (+0000 must be specified). If you include the x-amz-date header in your request, it overrides any value specified in the Date request header. When using AWS Signature Version 4, the x-amz-date header must be present in the signed request because the date header is not supported.
Support for Object Storage platform services
Object Storage platform services enable Object Storage tenant accounts to leverage external services such as a remote S3 bucket, a Simple Notification Service (SNS) endpoint, or an Elasticsearch cluster to extend the services provided by a grid.
The following table summarizes the available platform services and the S3 APIs used to configure them.
| Platform service | Purpose | S3 API used to configure the service |
|---|---|---|
| CloudMirror replication | Replicates objects from a source Object Storage bucket to the configured remote S3 bucket. | PUT Bucket replication |
| Notifications | Sends notifications about events in a source Object Storage bucket to a configured Simple Notification Service (SNS) endpoint. | PUT Bucket notification |
| Search integration | Sends object metadata for objects stored in a Object Storage bucket to a configured Elasticsearch index. | PUT Bucket metadata notification Note: This is a Object Storage custom S3 API. |
Testing your S3 REST API configuration
You can use the Amazon Web Services Command Line Interface (AWS CLI) to test your connection to the system and to verify that you can read and write objects to the system. Before you begin • You must have downloaded and installed the AWS CLI from aws.amazon.com/cli. • You must have created an S3 tenant account in the Object Storage system.
Steps
Configure the Amazon Web Services settings to use the account you created in the Object Storage system:
Enter configuration mode:
aws configure --profile ProfileNameEnter the AWS Access Key ID for the account you created.
Enter the AWS Secret Access key for the account you created.
Enter the default region to use, for example, us-east-1.
Enter the default output format to use, for example, JSON.
Create a bucket.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName create-bucket --bucket testbucket
- If the bucket is created successfully, the location of the bucket is returned, as seen in the following example:*
"Location": "/testbucket"
- Upload an object.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName put-object --bucket testbucket --key s3.pdf --body C:\s3-test\upload\s3.pdf
- List the contents of the bucket to verify that the object was uploaded.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName list-objects --bucket testbucket
- Delete the object.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName delete-object --bucket testbucket --key s3.pdf
- Delete the bucket.
aws s3api --endpoint-url https://api-object.bluvalt.com:8082 --profile ProfileName delete-bucket --bucket testbucket
How Object Storage implements the S3 REST API
A client application can use S3 REST API calls to connect to Storage Nodes and API Gateway Nodes, to create buckets, and to store and retrieve objects.
Object versioning
You can use versioning to retain multiple versions of an object, which protects against accidental deletion of objects, and enables you to retrieve and restore earlier versions of an object.
The Object Storage system implements versioning with support for most features, and with some limitations. Object Storage supports up to 1,000 versions of each object.
In Object Storage, object versioning can be combined with Information Lifecycle Management (ILM), rather than Amazon’s Object Lifecycle Management, which is not supported. You must explicitly enable versioning for each bucket to turn on this functionality for the bucket. Each object in your bucket is assigned a version ID which is generated by the Object Storage system.
Common request headers
The Object Storage system supports common request headers defined by the Simple Storage Service API Reference, with several exceptions.
| Request header | Implementation | |
|---|---|---|
| Authorization | •Full support for AWS Signature Version 2 •Support for AWS Signature Version 4, with the following exceptions: • The SHA256 value is not calculated for the body of the request. The user submitted value is accepted without validation. | |
| x-amz-copy-source-server-side- encryption-customer-key | Not implemented. | |
| x-amz-copy-source-server-side- encryption-customer-key-MD5 | Not implemented. | |
| x-amz-security-token | Not implemented. Returns XNotImplemented. | |
| x-amz-server-side-encryption- customer-algorithm | Not implemented. |
Common response headers
The Object Storage system supports all of the common response headers defined by the Simple Storage Service API Reference, with several exceptions
Authenticating requests
The Object Storage system supports both authenticated and anonymous access to objects using the S3 API.
The S3 API supports Signature version 2 and Signature version 4 for authenticating S3 API requests.
Authenticated requests must be signed using your access key ID and secret access key.
The Object Storage system supports two authentication methods: the HTTP Authorization header and using query parameters.
Using the HTTP Authorization header The HTTP Authorization header is used by all S3 API operations except Anonymous requests where permitted by the bucket policy. The Authorization header contains all of the required signing information to authenticate a request.
Using query parameters You can use query parameters to add authentication information to a URL. This is known as presigning the URL, which can be used to grant temporary access to specific resources. Users with the presigned URL do not need to know the secret access key in order to access the resource, which enables you to provide third-party restricted access to a resource.
Operations on buckets
The Object Storage system supports a maximum of 1,000 buckets for each S3 tenant account.
Bucket name restrictions follow the AWS US Standard region restrictions, but you should further restrict them to DNS naming conventions in order to support S3 virtual hosted-style requests.
The GET Bucket (List Objects) and GET Bucket versions operations support Object Storage consistency controls.
You can check whether updates to last access time are enabled or disabled for individual buckets.
The following table describes how Object Storage implements S3 REST API bucket operations. To perform any of these operations, the necessary access credentials must be provided for the account.
| Operation | Implementation | |
|---|---|---|
| DELETE Bucket | Implemented with all Amazon S3 REST API behavior. | |
| DELETE Bucket cors | This operation deletes the CORS configuration for the bucket. | |
| DELETE Bucket policy | This operation deletes the policy attached to the bucket. | |
| DELETE Bucket replication | This operation deletes the replication configuration attached to the bucket. | |
| GET Bucket (List Objects), version 1 | This operation returns some or all (up to 1,000) of the objects in a bucket. | |
| The Storage Class for objects is always listed as STANDARD, even if the object was ingested with the REDUCED_REDUNDANCY storage class option. | ||
| Requests that traverse a large number of keys require special handling. The request might return a truncated response or an empty response to avoid timing out. | ||
| To get the complete set of results, you need to continue making requests while updating the marker parameter, as you normally do with a truncated result. Always use NextMarker if it is present. In some cases, the Object Storage implementation of the S3 REST API returns a NextMarker, when the Amazon S3 REST API would not, because it is a better marker than the last key returned | ||
| GET Bucket acl | This operation returns a positive response and the ID, DisplayName, and Permission of the bucket owner, indicating that the owner has full access to the bucket. | |
| GET Bucket cors | This operation returns the cors configuration for the bucket. | |
| GET Bucket location | This operation returns the bucket’s region. By default, us- east-1 is returned unless a region was set using the LocationConstraint element in the PUT Bucket request. | |
| GET Bucket Object versions | With READ access on a bucket, this operation with the versions subresource lists metadata of all of the versions of objects in the bucket. | |
| GET Bucket notification | This operation returns the notification configuration attached to the bucket. | |
| GET Bucket policy | This operation returns the policy attached to the bucket. | |
| GET Bucket replication | This operation returns the replication configuration attached to the bucket. | |
| GET Bucket versioning | This implementation uses the versioning subresource to return the versioning state of a bucket. The versioning state returned indicates if the bucket is “Unversioned” or if the bucket is version “Enabled” or “Suspended.” | |
| HEAD Bucket | This operation determines if a bucket exists and you have permission to access it. | |
| PUT Bucket | This operation creates a new bucket. By creating the bucket, you become the bucket owner. | |
| • Bucket names must comply with the following rules: | ||
| o Must be unique across each Object Storage system (not just unique within the tenant account). | ||
| o Must be DNS compliant. | ||
| o Must contain between 3 and 63 characters. | ||
| o Can be a series of one or more labels, with adjacent labels separated by a period. Each label must start and end with a lowercase letter or a number and can only use lowercase letters, numbers, and hyphens. | ||
| o Must not look like a text-formatted IP address. | ||
| o Should not use periods in virtual hosted-style requests because periods will cause problems with server wildcard certificate verification. | ||
| • By | ||
| however, you can use the LocationConstraint request element in the request body to specify a different region. When using the LocationConstraint element, you must specify the exact name of a region that has been defined using the Grid Manager or the Grid Management API. Contact your system administrator if you do not know the region name you should use. | ||
| Note: An error will occur if your PUT Bucket request uses a region that has not been defined in Object Storage. | ||
| • When using a PUT Bucket request, you can include the SGCompliance XML element in the XML request body. SGCompliance is a custom Object Storage element that allows you to create buckets that are compliant. | ||
| default, buckets are created in the us-east-1 region; | ||
| PUT Bucket cors | This operation sets the CORS configuration for a bucket so that the bucket can service cross-origin requests. | |
| Cross-origin resource sharing (CORS) is a security mechanism that allows client web applications in one domain to access resources in a different domain. For example, suppose you use an S3 bucket named images to store graphics. By setting the CORS configuration for the images bucket, you can allow the images in that bucket to be displayed on the website http:// www.example.com. | ||
| PUT Bucket notification | This operation configures notifications for the bucket using the notification configuration XML included in the request body. | |
| You should be aware of the following implementation details: | ||
| • Object Storage supports Simple Notification Service (SNS) topics as destinations. Simple Queue Service (SQS) or Amazon Lambda endpoints are not supported. | ||
| • The destination for notifications must be specified as the URN of an Object Storage endpoint. Endpoints can be created using the Tenant Manager or the Tenant Management API. | ||
| The endpoint must exist for notification configuration to succeed. If the endpoint does not exist, a 400 Bad Request error is returned with the code InvalidArgument. | ||
| • Notifications on the s3:ReducedRedundancyLostObject event are not supported. | ||
| • Event notification messages use standard values for most keys, except for the following: | ||
| o eventSource returns sgws:s3 | ||
| o awsRegion: this key is not returned | ||
| o x-amz-id-2: this key is not returned | ||
| o arn returns urn:sgws:s3:::bucket-name | ||
| PUT Bucket policy | This operation sets the policy attached to the bucket. |
Operations on objects
This section describes how the Object Storage system implements S3 REST API operations for objects.
The following conditions apply to all object operations:
• All of the operations on objects, except GET Object ACL and OPTIONS /, support Object Storage consistency controls.
• Conflicting client requests, such as two clients writing to the same key, are resolved on a “latest- wins” basis. The timing for the “latest-wins”evaluation is based on when the Object Storage system completes a given request, and not on when S3 clients begin an operation.
• All objects in a Object Storage bucket are owned by the bucket owner, including objects created by an anonymous user, or by another account.
• Data objects ingested to the Object Storage system through Swift cannot be accessed through S3.
| Operation | Implementation |
|---|---|
| DELETE Object | Multi-Factor Authentication (MFA) and the response header x-amz- mfa are not supported. |
| Versioning | |
| To remove a specific version, the requestor must be the bucket owner and use the versionId subresource. Using this subresource permanently deletes the version. If the versionId corresponds to a delete marker, the response header x-amz-delete-marker is returned set to true. | |
| • If an object is deleted without the versionId subresource on a version enabled bucket, it results in the generation of a delete marker. The versionId for the delete marker is returned using the x-amz-version-id response header, and the x-amz-delete- marker response header is returned set to true. | |
| • If an object is deleted without the versionId subresource on a version suspended bucket, it results in a permanent deletion of an already existing ‘null’ version or a ‘null’ delete marker, and the generation of a new ‘null’ delete marker. The x-amz-delete- marker response header is returned set to true. | |
| DELETE Multiple Objects | Multi-Factor Authentication (MFA) and the response header x-amz- mfa are not supported. |
| Multiple objects can be deleted in the same request message. | |
| Note: The DELETE Multiple Objects request is not supported on versioned buckets. | |
| Unlike the PUT Object operation, the DELETE Multiple Objects operation does not support chunked transfer encoding and the content encoding gzip attributes. | |
| DELETE Object tagging | Uses the tagging subresource to remove all tags from an object. Implemented with all Amazon S3 REST API behavior. |
| Versioning | |
| If the versionId query parameter is not specified in the request, the operation deletes all tags from the most recent version of the object in a versioned bucket. | |
| GET Object | The following request headers are not supported and return |
| • x-amz-restore | |
| • x-amz-website-redirect-location | |
| Versioning | |
| If a versionId subresource is not specified, the operation fetches the most recent version of the object in a versioned bucket. If the current version of the object is a delete marker, a “Not Found” status is returned with the x-amz-delete-marker response header set to true. | |
| GET Object ACL | If the necessary access credentials are provided for the account, the operation returns a positive response and the ID, DisplayName, and Permission of the object owner, indicating that the owner has full access to the object. |
| GET Object tagging | Uses the tagging subresource to return all tags for an object. Implemented with all Amazon S3 REST API behavior |
| Versioning | |
| If the versionId query parameter is not specified in the request, the operation returns all tags from the most recent version of the object in a versioned bucket. | |
| HEAD Object | The following request headers are not supported and return XNotImplemented: |
| • x-amz-restore | |
| • x-amz-website-redirect-location | |
| Versioning | |
| If a versionId subresource is not specified, the operation fetches the most recent version of the object in a versioned bucket. If the current version of the object is a delete marker, a “Not Found” status is returned with the x-amz-delete-marker response header set to true. | |
| PUT Object tagging | Uses the tagging subresource to add a set of tags to an existing object. Implemented with all Amazon S3 REST API behavior Resolving conflicts |
| Conflicting client requests, such as two clients writing to the same key, are resolved on a “latest-wins” basis. The timing for the “latest- wins”evaluation is based on when the Object Storage system completes a given request, and not on when S3 clients begin an operation. | |
| Versioning | |
| If the versionId query parameter is not specified in the request, the operation add tags to the most recent version of the object in a versioned bucket. If the current version of the object is a delete marker, a “MethodNotAllowed” status is returned with the x-amz-delete- marker response header set to true. |
Error responses
The Object Storage system supports all standard S3 REST API error responses that apply. In addition, the Object Storage implementation adds several custom responses.
Supported S3 API error codes
| Name | HTTP status |
|---|---|
| AccessDenied | 403 Forbidden |
| BadDigest | 400 Bad Request |
| BucketAlreadyExists | 409 Conflict |
| BucketNotEmpty | 409 Conflict |
| IncompleteBody | 400 Bad Request |
| InternalError | 500 Internal Server Error |
| InvalidAccessKeyId | 403 Forbidden |
| InvalidArgument | 400 Bad Request |
| InvalidBucketName | 400 Bad Request |
| InvalidDigest | 400 Bad Request |
| InvalidEncryptionAlgorithmError | 400 Bad Request |
| InvalidPart | 400 Bad Request |
| InvalidPartOrder | 400 Bad Request |
| InvalidRange | 416 Requested Range Not Satisfiable |
| InvalidRequest | 400 Bad Request |
| InvalidStorageClass | 400 Bad Request |
| InvalidTag | 400 Bad Request |
| InvalidURI | 400 Bad Request |
| KeyTooLong | 400 Bad Request |
| MalformedXML | 400 Bad Request |
| MetadataTooLarge | 400 Bad Request |
| MethodNotAllowed | 405 Method Not Allowed |
| MissingContentLength | 411 Length Required |
| MissingRequestBodyError | 400 Bad Request |
| MissingSecurityHeader | 400 Bad Request |
| NoSuchBucket | 404 Not Found |
| NoSuchKey | 404 Not Found |
| NoSuchUpload | 404 Not Found |
| NotImplemented | 501 Not Implemented |
| NoSuchBucketPolicy | 404 Not Found |
| PreconditionFailed | 412 Precondition Failed |
| RequestTimeTooSkewed | 403 Forbidden |
| ServiceUnavailable | 503 Service Unavailable |
| SignatureDoesNotMatch | 403 Forbidden |
| TooManyBuckets | 400 Bad Request |
| UserKeyMustBeSpecified | 400 Bad Request |
Object Storage custom error codes |Name| Description| HTTP status | ——– | ——– | ——– | |XNoSuchBucketCompliance| The specified bucket does not have compliance enabled. | 404 Not Found | |XNotImplemented |The request you provided implies functionality that is not implemented. |501 Not Implemented |