AWS is Teaching New Tricks to S3

Recent Announcements from AWS for S3

Among the flurry of announcements at AWS re:invent in Dec 2016, the new features for S3 may have received little attention yet, but will be important in the longer term.

S3 is one of the first AWS services, and any refresh indicates that S3 still plays a vital role in Amazons comprehensive storage strategy among many other products. Apparently the AWS team found the time during their rush to new services to improve on the services which are already used a lot.

I took a look at the following items: (with my usual attention to security topics)

  • Price reductions
  • New Console
  • New Storage Gateway
  • Storage Management
    • Operations: S3 Cloud Watch Metrics
    • S3 Analytics
  • Compliance Management
    • CloudTrail S3 Data Events
    • S3 Object Tags
    • S3 Inventory

Price reduction

AWS announced price reductions for S3 of 23% and more on 21 Nov 2016.
Heavy users of S3 recently have not been very happy about the costs of S3. The last good news was the introduction of the infrequent access (IA) option in Sep 2015, at somewhat lower cost if you optimized the associated management. The last price reduction for S3 was in March 2014, whereas raw hard drive capacity is now costing half of what it cost 32 months ago.

New Console

The new console is available to everyone: Choose ’Opt In’ in the S3 start screen (you can switch back).
Value:

  • It looks much nicer, and only the new console shows the new features.
  • One obvious improvement is that the start screen already shows the location of your buckets. This is reassuring if you have to watch it for legal reasons, and new buckets are created in your dynamic organization frequently. With the old console people may have bypassed the “location” setting without noticing.

Caution:

  • Other parts have a new look, but the UX (user experience) did not improve.
  • The permissions tab prominently shows ACLs, which you should not use.
  • Bucket policy must be selected via a drop-down box.
  • The policy editor opens in a new look but with the charm of a text terminal.
  • There is no indication of IAM policies which may affect the bucket.

New Storage Gateway

There is a new file gateway to S3, which you can install on your premises to act as an NFS mount point. This way your users can move files into and from S3 via NFS. As there are 3rd party file system drivers already, you would probably evaluate it for performance and quality. Amazon positioned it as a tool for migrating data into the cloud more easily.

Storage Management: S3 Analytics

S3 Analytics helps you understand your storage usage. It should not be mixed up with the also announced Amazon Athena, which queries data stored in S3.

S3 Analytics shows aggregated information about how much storage is used, is read, etc. Especially it shows how much data is not read for a long time, i.e. not read for 60 days, and can thus be moved to IA (infrequent access) storage class, which saves money.

It thereby helps you work with LifeCycle rules, which existed for some time, but previously you had to guess when to move data, with S3 Analytics you can make a more informed decision.

Issue was that migration to Glacier is in many unatrictive, because retrieval may take hours.

Archival to Glacier was hard to retrieve.

Value:

  • Free metrics on BucketSizeBytes, NumberOfObjects, DataTransfer per age interval
  • Cost optimization potential by better decision to move to IA (infrequent access)

Caution:

  • Some metrics have a cost per object count
  • Set appropriate permissions on the GetAnalytics API
  • There was no mention of reporting on encryption

Storage Management: Cloud Watch Metrics

CloudWatch will deliver operational metrics on S3: Errors, Bytes Downloaded, Request Latency, etc. This is probably a welcome addition if your team already uses Cloud Watch and your operations depends on S3 performance.

Compliance Management: CloudTrail S3 Data Events

Previously CloudTrail only recorded events on bucket level, now it can also record object level events like GetObject or PutObject.

Value:

  • CloudTrail is guaranteed to be complete.
    Previously you had to use S3 access logs, which were not guaranteed to be complete.

Caution:

  • Do not enable it for all GetObject or PutObject calls in all buckets
    it costs $1 per million events + storage, the usual access logs only cost storage

Compliance Management: S3 Object Tags

Tagging is quite widely used in EC2, is popular in AWS, and can now also be used with S3.
In essence it is a very simple mechanism for users to include their meta data as (key, value) pairs, up to 10 tags per object, besides the static meta data which are attached to each S3 object.

Value:

  • Classification of assets can make compliance tangible and practical
  • Access policies can depend on the meta data
  • Lifecycle rules can be controlled

Caution:

  • Management of your tags (creation, change, access control) should be planned
  • Changing tags on a large scale will require code (CLI or Lambda)
  • Price: $0.01 per 10.000 tags per month

Compliance Management: S3 Inventory

S3 Inventory delivers a snapshot of detailed information about each object in an s3 bucket.

Value:

  • You can maintain an index of files, which may be necessary for certain use cases.
  • Meta data include useful fields, e.g. replication status.

Caution:

  • The inventory takes some time and is delivered with a batch job to another S3 bucket, either daily or weekly. You will be notified when it is complete via SNS, e.g. by email.
  • Actually, I was notified by an AWS Support Request in my inbox:
    “You need to setup permissions for the delivery of the logs”.
    Better you setup permissions right away.
  • The metadata do not contain information about encryption status

Summary

The addition of new features to S3 is good news, because it demonstrates that AWS cares about the business case of their long-time users.

The new features focus on storage management. Heavy users of the service can now more easily optimize to lower their cost. Along with the price reduction this will keep this audience happy for a longer time.

For organizations which do not yet have a high monthly bill of S3, the new features are mostly a nice-to-have. They should take a closer look at tagging and inventory, which can help with compliance requirements.

Take also a look at the accompanying changes to Glacier, which go into the same direction.

For the security geeks there was not much new.

Ekkard Schnedermann, https://www.schnedermann.com, 15 Dec 2016

Sources

Webinar “Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016”: Recording (https://www.youtube.com/watch?v=xZ_Tch87mXw): https://www.youtube.com/watch?v=xZ\_Tch87mXw
Webinar “Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016”: Recording: https://www.youtube.com/watch?v=xZ_Tch87mXw;
Slides: (http://www.slideshare.net/AmazonWebServices/deep-dive-on-s3-storage-management-covering-new-feature-announcements-december-2016-monthly-webinar-series): http://www.slideshare.net/AmazonWebServices/deep-dive-on-s3-storage-management-covering-new-feature-announcements-december-2016-monthly-webinar-series