AWS S3 Storage

Updated: at 06:22 AM

Table of content

S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can use Amazon S3 to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides management features so that you can optimize, organize, and configure access to your data to meet your specific business, organizational, and compliance requirements.

Features

FeatureDescription
High DurabilityS3 provides 99.999999999% durability for objects by storing copies in multiple systems across at least three AZs.
ScalabilityS3 can store an unlimited amount of data, from small files to large datasets, scaling automatically as storage needs grow.
SecurityOffers comprehensive security features like encryption in transit and at rest, IAM policies, S3 Bucket Policies, and Access Control Lists.
Data Lifecycle ManagementAutomatically moves objects between different storage classes and archives or deletes data based on defined rules.
VersioningKeeps multiple versions of an object in the same bucket, allowing you to preserve, retrieve, and restore every version.
Cross-Region Replication (CRR)Automatically replicates data across AWS regions for enhanced availability and compliance.
Event NotificationsSends notifications when specified events occur in your bucket, using SNS, SQS, or Lambda.
Storage Class AnalysisMonitors access patterns and suggests when to move data to more cost-effective storage classes.
S3 Select and Glacier SelectRetrieve only a subset of data from an object by using simple SQL expressions.
Transfer AccelerationSpeeds up the transfer of files by utilizing Amazon CloudFront’s globally distributed edge locations.
Multipart UploadAllows large files to be uploaded in smaller parts concurrently, improving throughput and the ability to resume uploads.
Object LockProvides WORM (Write Once Read Many) capability to prevent object deletion or modification for a fixed amount of time.

Other important features are:

Storage Classes

Each object in Amazon S3 has a storage class associated with it. For example, if you list the objects in an S3 bucket, the console shows the storage class for all the objects in the list. Amazon S3 offers a range of storage classes for the objects that you store. You choose a class depending on your use case scenario and performance access requirements. All of these storage classes offer high durability.

Storage ClassUse Case ExamplesAvailabilityDurabilityMinimum Storage DurationMinimum Billable Object SizeRetrieval FeeFirst Byte Latency
S3 StandardFrequently accessed data, general-purpose99.99%99.999999999%NoneNoneNoMilliseconds
S3 Intelligent-TieringData with unknown or changing access patterns99.9%99.999999999%None128KBNo (for frequent access tier)Milliseconds
S3 Standard-IALong-lived, infrequently accessed data99.9%99.999999999%30 days128KBYesMilliseconds
S3 One Zone-IAInfrequently accessed data, not requiring multiple AZ data resilience99.5%99.999999999%30 days128KBYesMilliseconds
S3 Glacier Instant RetrievalArchive data with rapid retrieval99.9%99.999999999%90 days128KBYesMilliseconds to seconds
S3 Glacier Flexible Retrieval (formerly Glacier)Archive data accessed once or twice a year99.99%99.999999999%90 days128KBYesMinutes to hours
S3 Glacier Deep ArchiveLong-term archive, accessed very infrequently99.9%99.999999999%180 days128KBYes12 hours (standard), 48 hours (bulk)

Setting the storage class of an object

Object Lifecycle Management

Access Control List (ACL)

{
  "Owner": {
    "DisplayName": "awspetsafemasterdevelop",
    "ID": "4fccb070bc8890e265d43" // creater or called Canonical User Id
  },
  "Grants": [
    {
      "Grantee": {
        "DisplayName": "awspetsafemasterdevelop",
        "ID": "4fccb070bc8890e265d43", //default assigned to creater
        "Type": "CanonicalUser"
      },
      "Permission": "FULL_CONTROL"
    }
  ]
}

Canned ACL & Grants

You grant permissions to objects (max 100 grants). To grant you can either use CannedACL or define explicitly. You can not do both.

  1. Canned ACL: S3 supports a set of pre-defined grants, known as canned ACLs. Each ACL has a predefined set of grantees and permissions.
  1. Set Explicitly - You can use x-amz-acl in header of the request to set it
curl ... -H "x-amz-acl:private" https://mybucket.s3.amazonaws.com/test.json

Pre-signed URL

Share Object with Others Temporarily

aws s3 presign s3://mybucket/myimage.png

# returns
https://mybucket.s3.us-east-1.amazonaws.com/myimage.png?
    X-Amz-Algorithm=AWS4-HMAC-SHA256&
    X-Amz-Credential=ASIAXQYGD74PCZPYL37D%2F20210528%2Fus-east-1%2Fs3%2Faws4_request&
    X-Amz-Date=20210528T104921Z&
    X-Amz-Expires=3600&
    X-Amz-SignedHeaders=host&
    X-Amz-Security-Token=IQoJb3...D&
    X-Amz-Signature=1753...

Requester Pays

Use-Cases

S3 or EFS

Customer Encryption

Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C) for the S3 bucket ensures data security both at rest and in-transit.

  1. For Amazon S3 REST API calls, you have to include the following HTTP Request Headers:
  1. For Presigned URLs, you should specify the algorithm using the x-amz-server-side-encryption-customer-algorithm request header.