Simple Storage Service (S3)

S3 provides long-term shared storage for research data and results.

Buckets, objects, and prefixes

An S3 bucket stores objects. An object consists of data, metadata, and a unique key such as users/demonstration/file.txt.

S3 does not have a traditional directory hierarchy. The / characters are part of an object's key, and strings such as users/demonstration/ are called prefixes. The AWS Console and CLI present prefixes like folders because that is a convenient way to browse them.

The lab's research bucket is:

rprlab-research-376129434881-eu-north-1-an

Open it under S3 > Buckets > rprlab-research-376129434881-eu-north-1-an in the AWS Management Console.

Where data belongs

Prefix	Use it for	Retention expectation
`projects/<project>/`	Project inputs, outputs, documentation, and reproducibility material	Retained according to the project's needs
`shared/`	Resources used across projects	Retained while they remain useful
`shared/databases/`	Shared reference databases such as InterPro	Maintained centrally; avoid duplicate copies
`users/<username>/`	Personal transfers, temporary files, and configuration	Temporary; reviewed first during cleanup and removed when a user leaves

Do not use users/<username>/ as the only location for important project data. Move retained material into the appropriate project or shared prefix.

Using the AWS CLI

Install and configure the AWS CLI as described under Accessing AWS. The examples below use the shared users/demonstration/ prefix.

Access is not ownership

Your permissions may allow you to change objects outside your personal prefix. Modify shared or project data only when you understand the effect and have coordinated with the people using it.

Listing objects

List the top level of the demonstration prefix:

shell

aws s3 ls "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/"

List everything under the demonstration prefix:

shell

aws s3 ls "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/" \
    --recursive

Uploading files

Upload a file while keeping its current name:

shell

aws s3 cp \
    input.txt \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/"

Upload the contents of a directory:

shell

aws s3 cp \
    ./input-folder/ \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/input-folder/" \
    --recursive

The source directory's contents are copied beneath the destination prefix. Include input-folder/ in the destination, as above, when you want that name represented in S3.

Downloading

Download a single object into the current directory:

shell

aws s3 cp \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/output.txt" \
    .

Download everything beneath a prefix:

shell

aws s3 cp \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/output-folder/" \
    ./output-folder/ \
    --recursive

Synchronizing directories

Use sync to transfer only files that are new or have changed.

Upload local changes:

shell

aws s3 sync \
    ./input-folder/ \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/input-folder/"

Download changes from S3:

shell

aws s3 sync \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/output-folder/" \
    ./output-folder/

These commands do not delete files that exist only at the destination. The optional --delete flag does; use it only after checking the command with --dryrun and confirming that destination-only files should be removed.

Moving and deleting data

Do not assume deletion is recoverable

S3 does not protect against deletion or overwrite. Confirm the complete source and destination paths before running mv, rm, sync --delete, or any recursive command.

An S3-to-S3 move copies each source object to its new key and then deletes the source. Moving a large prefix can therefore take time and generate many copy requests that add up costs. Prefer stable project prefixes so large datasets do not need to be reorganized repeatedly.

Move one object:

shell

aws s3 mv \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/input.txt" \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/output.txt"

Delete one object:

shell

aws s3 rm \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/output.txt"

Delete everything beneath a prefix:

shell

aws s3 rm \
    "s3://rprlab-research-376129434881-eu-north-1-an/users/demonstration/scratch/output-folder/" \
    --recursive

Add --dryrun to a recursive cp, mv, rm, or sync command first when you are uncertain which objects it will affect.

Practical considerations

S3 is object storage, not a mounted working disk. Applications generally cannot edit part of an object in place; an updated file is uploaded as a new object at that key.
Keep active scratch data on EBS. Upload durable inputs and outputs to S3 before terminating the EC2 instance.
Prefer clear, stable keys using project names and meaningful file names. Avoid repeatedly moving large datasets between prefixes.
Confirm important uploads with aws s3 ls or a download test before deleting the local copy or terminating an instance.
Storage, API requests, retrieval from some storage classes, and some data transfers can incur charges. Large collections of small files generate more requests than a smaller number of larger archives. See Amazon S3 pricing for current details.