Object Storage - Get started

First steps

Create an Object Store

Connect to ITCare and access the Storage section from the main menu on the left. Click on "Create an Object Store"

Select The "Data Center" (also named Region) that will own your Object Store:

When Geo-Replication is Enabled, Objects are available on both endpoint.

The Geo Replication can't be Enabled or Disabled once the Object Store is created.

Search and select a "Global Service":

Enter a name to your Object Store (see Limitation & Best Practices#Limitations)

You can also set a "Quota". The quota can be changed at any time after the creation of the Object Store.

Your Object Store name will be prefixed by "cos" + the Cloud Name of the service global selected : cos-<cloud_name>-<Your Object Store name>

Example: cos-cegedimit-hello

The last step the summary of your Object Store creation request.

You check if information are correct. Click on the "Submit" button to launch the creation.

Once the creation is done (it can take few minutes), a pop-up appear, displaying your credentials and available endpoints for your Object Store:

Credentials

• User Name → is your access_key

• Password → is your secret_key

Keep your secret_key safe, it will not be displayed anymore.

You have the possibility to regenerate, see Manage Object Users.

Endpoints

• If you selected a Data center with Geo-Replication enabled (Step 2), you will have 2 endpoints, one for each Data Centers.

• If you selected a Data center with Geo-Replication disabled (Step 2), you will have only 1 endpoint, corresponding to the Data Center selected.

Manage an Object Store

This page show detailed information about your Object Store:

• The service global which the Object Store is part of

• The data center where the Object Store is located

• Global size and numbers of object

• Quota status

• Object users

You also to possibility to manage:

• Quota

• Object Users

• Delete the Object Store

Create a Bucket

You have created your Object Store, it's time now to create a Bucket:

We use aws s3 and aws s3api command line tools from AWSCLIv2 on Linux.

${S3_ENDPOINT} and${S3_PROFILE} are environment variables.

Use the command "mb" to create a Bucket:

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 mb s3://my-bucket

# Output
make_bucket: my-bucket

List Buckets :

# List buckets
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls
 
# Output
2022-11-13 11:43:54 my-bucket

Upload an Objet

We have now a Bucket, let's upload objects in it:

We use aws s3 and aws s3api command line tools from AWSCLIv2 on Linux.

${S3_ENDPOINT} and${S3_PROFILE} are environment variables.

Use the command "cp" to upload an object:

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 cp feather.ttf s3://my-bucket
# Output
upload: ./feather.ttf to s3://my-bucket/feather.ttf
 
# List content of the bucket
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls s3://my-bucket
 
# Output
2022-11-13 11:47:42 81512 feather.ttf

You can specify a prefix when you upload an object to a Bucket:

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 cp feather.ttf s3://my-bucket/prefix-1/prefix-2/
 
# Output
upload: ./feather.ttf to s3://my-bucket/prefix-1/prefix-2/feather.ttf
 
# List content of the bucket
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls s3://my-bucket/
 
# Output
PRE prefix-1/
 
# List content of the bucket at prefix-1
aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 ls s3://my-bucket/prefix-1/
 
# Output
PRE prefix-2/

Manage Object Store Quota

You can specified a Quota on your Object Store in order to limit space used.

When the Quota is reached, upload in buckets in the Object Store are denied.

To manage Quota, go the detailed information page of your Object Store and click on the Manage quota button.

You can set Quota from 1Gb up to 8Tb.

If you need more than 8Tb Quota, please contact cegedim.cloud.

Once Quota applied, on can you follow the status of the Quota on the detailed information page of your Object Store:

When the Quota limit is reached, upload is denied (HTTP 403 Forbidden):

aws --endpoint-url=${S3_ENDPOINT} --profile=${S3_PROFILE} s3 cp s3://my-bucket/hello/

Output

upload failed: ./feather.ttf to s3://my-bucket/hello/feather.ttf An error occurred (Forbidden) when calling the PutObject operation: Check if quota has been exceeded or object has too many versions

Recommended Clients

S3Browser

S3 Browser is a freeware for Windows desktops (only). It offers basic functionalities with free version. For advanced features, the pro version must be acquired.

S3 Browser - Amazon S3 Client for Windows. User Interface for Amazon S3. S3 Bucket Explorer.

AWS CLI

AWS CLI is the official AWS command line interface. It offers all functionalities and best performance to use with cegedim.cloud Object Storage Service.

Command Line Interface - AWS CLI - AWSAmazon Web Services, Inc.

S5cmd

s5cmd is an alternative to aws cli, is a very fast S3 and local filesystem execution tool.

GitHub - peak/s5cmd: Parallel S3 and local filesystem execution tool.GitHub

Software Development Kit (SDK)

We recommend to use official AWS SDK:

• Java : https://aws.amazon.com/documentation/sdkforjava/

• .NET : https://aws.amazon.com/documentation/sdkfornet/

• Node.js : https://aws.amazon.com/documentation/sdk-for-javascript/

• Python : https://boto3.readthedocs.org/en/latest/

Developing with Amazon S3 using the AWS SDKs, and explorers - Amazon Simple Storage ServiceAmazon Simple Storage Service

Resilience and DRP

If you are using a Geo-Replicated Object Store, your objects will be available on both regions of cegedim.cloud Object Storage Service.

Best practice would be to use a client with built-in capability to switch over the two regions, and then:

• Automatically fail-over to second region when primary is down

• Have no configuration to do to enable fail-over of your application in another region

Java pseudo code

/**
  Class responsible to deliver a valid s3 client to other components of application.
  It holds s3 client instances and has ability to test if client is available for each region.
**/
class S3ClientFactory {
    private List<AmazonS3Client> clients;
 
    public void init() {
        // initialization of your clients here
    }
 
    /**
    returns a valid s3 client or throw exception
    **/
    public AmazonS3Client getClient() throws ServiceUnavailableException {
        for (AmazonS3Client client : clients) {
            if (client.isAvailable()) {
                return client;
            }
        }
        throw new ServiceUnavailableException("No S3 client is currently available");
    }
 
}
 
class MyApplication {
     
    S3ClientFactory factory = new S3ClientFactory();
 
    // get a client, whatever the region is
    AmazonS3Client client = factory.getClient();
 
    client.putObject("mybucket", "path/to/object", "One more");
 
}

Emptying a Bucket

• Emptying a Bucket is irreversible.

• Deleted Objects or Buckets can't be restored.

• Delete operation can take time depending on the number of objects and versions stored in the bucket.

Using S3 CLI

You can use any S3 CLI tools, like AWS CLI, s3cmd or s3browser.

You can empty a bucket using this kind of tools only if the bucket does not have "Versioning enabled". (See Manage versioning in Bucket)

If versioning is not enabled, you can use the rm (remove) command with the --recursive parameter to empty the bucket (or remove a subset of objects with a specific key name prefix).

The following rm command removes objects that have the key name prefix doc, for example, doc/doc1 and doc/doc2.

$ aws s3 rm s3://bucket-name/doc --recursive

Use the following command to remove all objects without specifying a prefix.

$ aws s3 rm s3://bucket-name --recursive

You can't remove objects from a bucket that has versioning enabled. S3 adds a delete marker when you delete an object, which is what this command does.

See Bucket Lifecycle for more information.

You can't remove objects with Object Lock enable until the retention period defined is reached.

Using a lifecycle configuration

If you use a lifecycle configuration to empty your bucket, the lifecycle configuration should include:

• current versions

• non-current versions

• delete markers

• incomplete multipart uploads

You can add lifecycle configuration rules to expire all objects or a subset of objects that have a specific key name prefix. For example, to remove all objects in a bucket, you can set a lifecycle rule to expire objects one day after creation.

If your bucket has versioning enabled, you can also configure the rule to expire non-current objects.

To fully empty the contents of a versioning enabled bucket, you will need to configure an expiration policy on both current and non-current objects in the bucket.

You can add lifecycle policy on the bucket using AWS CLI or GUI Client like s3browser.

For more information about lifecycle configuration, see Bucket Lifecycle.

Find below some lifecycle policies to empty bucket:

Non versioning Bucket

{
    "Rules": [
        {
            "Expiration": {
                "Days": 1
            },
            "ID": "lifecycle-v2-expire-current-and-mpu",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        }
    ]
}

Versioning Bucket

{
    "Rules": [
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "lifecycle-v2-expire-non-current-and-dmarkers-and-mpu",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        }
    ]
}

Both versioning and non versioning Bucket

Below an example of lifecycle configuration to emptying a Bucket, handle versioning Enable and Disable:

{
    "Rules": [
        {
            "Expiration": {
                "Days": 1
            },
            "ID": "lifecycle-v2-purge-all",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled"
        }
    ]
}