Object Storage - Get started

First steps

Create an Object Store

Connect to ITCare and access the Storage section from the main menu on the left. Click on "Create an Object Store"

Select The "Data Center" (also named Region) that will own your Object Store:

When Geo-Replication is Enabled, Objects are available on both endpoint.

The Geo Replication can't be Enabled or Disabled once the Object Store is created.

Search and select a "Global Service":

Enter a name to your Object Store (see Limitation & Best Practices#Limitations)

You can also set a "Quota". The quota can be changed at any time after the creation of the Object Store.

Your Object Store name will be prefixed by "cos" + the Cloud Name of the service global selected : cos-<cloud_name>-<Your Object Store name>

Example: cos-cegedimit-hello

The last step the summary of your Object Store creation request.

You check if information are correct. Click on the "Submit" button to launch the creation.

Once the creation is done (it can take few minutes), a pop-up appear, displaying your credentials and available endpoints for your Object Store:

Credentials

• User Name → is your access_key

• Password → is your secret_key

Keep your secret_key safe, it will not be displayed anymore.

You have the possibility to regenerate, see Manage Object Users.

Endpoints

• If you selected a Data center with Geo-Replication enabled (Step 2), you will have 2 endpoints, one for each Data Centers.

• If you selected a Data center with Geo-Replication disabled (Step 2), you will have only 1 endpoint, corresponding to the Data Center selected.

Manage an Object Store

This page show detailed information about your Object Store:

• The service global which the Object Store is part of

• The data center where the Object Store is located

• Global size and numbers of object

• Quota status

• Object users

You also to possibility to manage:

• Quota

• Object Users

• Delete the Object Store

Create a Bucket

You have created your Object Store, it's time now to create a Bucket:

We use aws s3 and aws s3api command line tools from AWSCLIv2 on Linux.

${S3_ENDPOINT} and${S3_PROFILE} are environment variables.

Use the command "mb" to create a Bucket:

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 mb s3://my-bucket

# Output
make_bucket: my-bucket

List Buckets :

# List buckets
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls
 
# Output
2022-11-13 11:43:54 my-bucket

Upload an Objet

We have now a Bucket, let's upload objects in it:

We use aws s3 and aws s3api command line tools from AWSCLIv2 on Linux.

${S3_ENDPOINT} and${S3_PROFILE} are environment variables.

Use the command "cp" to upload an object:

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 cp feather.ttf s3://my-bucket
# Output
upload: ./feather.ttf to s3://my-bucket/feather.ttf
 
# List content of the bucket
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls s3://my-bucket
 
# Output
2022-11-13 11:47:42 81512 feather.ttf

You can specify a prefix when you upload an object to a Bucket:

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 cp feather.ttf s3://my-bucket/prefix-1/prefix-2/
 
# Output
upload: ./feather.ttf to s3://my-bucket/prefix-1/prefix-2/feather.ttf
 
# List content of the bucket
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls s3://my-bucket/
 
# Output
PRE prefix-1/
 
# List content of the bucket at prefix-1
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls s3://my-bucket/prefix-1/
 
# Output
PRE prefix-2/

Manage Object Store Quota

You can specified a Quota on your Object Store in order to limit space used.

When the Quota is reached, upload in buckets in the Object Store are denied.

To manage Quota, go the detailed information page of your Object Store and click on the Manage quota button.

You can set Quota from 1Gb up to 8Tb.

If you need more than 8Tb Quota, please contact cegedim.cloud.

Once Quota applied, on can you follow the status of the Quota on the detailed information page of your Object Store:

When the Quota limit is reached, upload is denied (HTTP 403 Forbidden):

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 cp s3://my-bucket/hello/

Output

upload failed: ./feather.ttf to s3://my-bucket/hello/feather.ttf An error occurred (Forbidden) when calling the PutObject operation: Check if quota has been exceeded or object has too many versions

Recommended Clients

S3Browser

S3 Browser is a freeware for Windows desktops (only). It offers basic functionalities with free version. For advanced features, the pro version must be acquired.

S3 Browser - Amazon S3 Client for Windows. User Interface for Amazon S3. S3 Bucket Explorer.

AWS CLI

AWS CLI is the official AWS command line interface. It offers all functionalities and best performance to use with cegedim.cloud Object Storage Service.

Command Line Interface - AWS CLI - AWSAmazon Web Services, Inc.

S5cmd

s5cmd is an alternative to aws cli, is a very fast S3 and local filesystem execution tool.

GitHub - peak/s5cmd: Parallel S3 and local filesystem execution tool.GitHub

Software Development Kit (SDK)

We recommend to use official AWS SDK:

• Java : https://aws.amazon.com/documentation/sdkforjava/

• .NET : https://aws.amazon.com/documentation/sdkfornet/

• Node.js : https://aws.amazon.com/documentation/sdk-for-javascript/

• Python : https://boto3.readthedocs.org/en/latest/

Developing with Amazon S3 using the AWS SDKs, and explorers - Amazon Simple Storage ServiceAmazon Simple Storage Service

Resilience and DRP

If you are using a Geo-Replicated Object Store, your objects will be available on both regions of cegedim.cloud Object Storage Service.

Best practice would be to use a client with built-in capability to switch over the two regions, and then:

• Automatically fail-over to second region when primary is down

• Have no configuration to do to enable fail-over of your application in another region

Java pseudo code

/**
  Class responsible to deliver a valid s3 client to other components of application.
  It holds s3 client instances and has ability to test if client is available for each region.
**/
class S3ClientFactory {
    private List<AmazonS3Client> clients;
 
    public void init() {
        // initialization of your clients here
    }
 
    /**
    returns a valid s3 client or throw exception
    **/
    public AmazonS3Client getClient() throws ServiceUnavailableException {
        for (AmazonS3Client client : clients) {
            if (client.isAvailable()) {
                return client;
            }
        }
        throw new ServiceUnavailableException("No S3 client is currently available");
    }
 
}
 
class MyApplication {
     
    S3ClientFactory factory = new S3ClientFactory();
 
    // get a client, whatever the region is
    AmazonS3Client client = factory.getClient();
 
    client.putObject("mybucket", "path/to/object", "One more");
 
}

Emptying a Bucket

• Emptying a Bucket is irreversible.

• Deleted Objects or Buckets can't be restored.

• Delete operation can take time depending on the number of objects and versions stored in the bucket.

Using S3 CLI

You can use any S3 CLI tools, like AWS CLI, s3cmd or s3browser.

You can empty a bucket using this kind of tools only if the bucket does not have "Versioning enabled". (See Manage versioning in Bucket)

If versioning is not enabled, you can use the rm (remove) command with the --recursive parameter to empty the bucket (or remove a subset of objects with a specific key name prefix).

The following rm command removes objects that have the key name prefix doc, for example, doc/doc1 and doc/doc2.

$ aws s3 rm s3://bucket-name/doc --recursive

Use the following command to remove all objects without specifying a prefix.

$ aws s3 rm s3://bucket-name --recursive

You can't remove objects from a bucket that has versioning enabled. S3 adds a delete marker when you delete an object, which is what this command does.

See Bucket Lifecycle for more information.

You can't remove objects with Object Lock enable until the retention period defined is reached.

Using a lifecycle configuration

If you use a lifecycle configuration to empty your bucket, the lifecycle configuration should include:

• current versions

• non-current versions

• delete markers

• incomplete multipart uploads

You can add lifecycle configuration rules to expire all objects or a subset of objects that have a specific key name prefix. For example, to remove all objects in a bucket, you can set a lifecycle rule to expire objects one day after creation.

If your bucket has versioning enabled, you can also configure the rule to expire non-current objects.

To fully empty the contents of a versioning enabled bucket, you will need to configure an expiration policy on both current and non-current objects in the bucket.

You can add lifecycle policy on the bucket using AWS CLI or GUI Client like s3browser.

For more information about lifecycle configuration, see Bucket Lifecycle.

Find below some lifecycle policies to empty bucket:

Non versioning Bucket

{
    "Rules": [
        {
            "Expiration": {
                "Days": 1
            },
            "ID": "lifecycle-v2-expire-current-and-mpu",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        }
    ]
}

Versioning Bucket

{
    "Rules": [
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "lifecycle-v2-expire-non-current-and-dmarkers-and-mpu",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        }
    ]
}

Both versioning and non versioning Bucket

Below an example of lifecycle configuration to emptying a Bucket, handle versioning Enable and Disable:

{
    "Rules": [
        {
            "Expiration": {
                "Days": 1
            },
            "ID": "lifecycle-v2-purge-all",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled"
        }
    ]
}

Limitation and Best Practices

Object Stores

Limitations

The following rules apply to the naming of Object Stores in cegedim.cloud Object Storage Service:

• Must be between one and 255 characters in length

• Can include hyphen (-) and alphanumeric characters ([a-zA-Z0-9])

• Avoid the use of underscores (_)

• Avoid the use of UPPERCASE letters

• Cannot start with a dot (.)

• Cannot contain a double dot (..)

• Cannot end with a dot (.)

• Cannot contain spaces

• Must not be formatted as IPv4 address

• Object Store names must be unique in cegedim.cloud Object Storage Service

Best Practices

• Create Object Store per Business Unit or per application

• Geo Replication can't be enable or disable once the Object Store created

• For best performance, recommended to have less than 1000 buckets in a single Object Store

• Object Stores should be DNS compatible

Buckets

Limitations

The following rules apply to the naming of S3 buckets in cegedim.cloud Object Storage Service:

• Must be between 3 and 255 characters in length.

• Can include dot (.), hyphen (-), and underscore (_) characters and alphanumeric characters ([a-zA-Z0-9])

• Avoid the use of UPPERCASE letters

• Can start with a hyphen (-) or alphanumeric character

• Cannot start with a dot (.)

• Cannot contain a double dot (..)

• Cannot end with a dot (.)

• Cannot contain spaces

• Must not be formatted as IPv4 address

• Bucket names must be unique within an Object Store

Best Practices

• Use buckets for specific environment, workflow, or uses. For instance: dev, test, finance, operations, etc

• In an Object Store with the Geo Replication enabled, create buckets using the closest (EB4 or ET1) endpoint to the application accessing and updating the objects

There is overhead involved with checking the latest copy if the ownership of object is at a remote endpoint

• For best performance, recommended to have less than 1000 buckets in a single Object Store

• Bucket names should be DNS compatible

Objects

Limitations

The following rules apply to the naming of Objects in cegedim.cloud Storage Service:

• Cannot be null or an empty string

• Length range must be between 1 and 255 (Unicode char)

• Avoid using space

• No validation on characters

Best Practices

• Object names should be DNS compatible

Small Objects vs Large Objects

This section provides useful tips when handling small and large objects within your application. It also provides some information on cegedim.cloud Object Storage Service versioning and compression details and options.

Small Objects

A small object is considered to be an object that is less than 100 KB.

cegedim.cloud Object Storage Service has a special internal mechanism which helps performance for data writes of small objects. It aggregates multiple small data objects queued in memory and then

write them in a single disk operation, up to 2MB of data. This improves performance by reducing the number of round-trips to process individual writes to storage.

Although cegedim.cloud Object Storage Service has optimizations for small writes, if there is an option in your application to define a size

then choose a larger size (e.g. 1 MB rather than 64 KB) or a value that aligns to the cegedim.cloud Object Storage Service internal buffer size of 2 MB for performance.

Large Objects

One of the issues with reading and writing large objects is performance.

cegedim.cloud Object Storage Service provides certain API features to reduce the impact on performance for large objects, such as, multipart uploads. Some tips to alleviate some of the issues for large object access include:

When working with large objects (> 100 MB), utilize the multipart upload feature. This allows pause and resume uploads for large objects.

cegedim.cloud Object Storage Service internal buffer size is 2 MB. For size inferior to 1 GB, use multiple of 2 MB (e.g. 8 MB).

cegedim.cloud Object Storage Service chunk size is 128MB. For size superior to 1 GB, use 128 MB part size.

Performance throughput can be improved by parallelizing uploads within your application.

Use APIs that allows for easy upload and download, for instance:

• In Java, use the TransferManager

• In .NET, use TransferUtility