How do I use Second Tier Storage from the command line?
To access the MSI second-tier storage, each user has a set of s3 credentials. The s3 credentials act like a username and password to control access to the storage behind our S3 gateway server s3.msi.umn.edu.
Command-line Access
The default command-line tool is s3cmd. This command line tool provides access to the full range of S3 features, including creating, deleting, and managing the permissions of buckets and files stored in buckets. A bucket is a container for all the files (AKA objects) that you store on the Second-tier storage systems. You can think of a bucket in the same way that you think of a top level file folder. The full manual for s3cmd is available from its homepage and below are some examples of the most commonly used s3cmd features.
Note that the external s3cmd documentation includes some features that only apply to Amazon's own S3 service.
Also note that on MSI systems we "wrap" the s3cmd command so that it is configured automatically with the MSI tier2 endpoint and your tier2 credentials - so you can skip the s3cmd documentation on configuration.
To create your own bucket (AKA folder) on s3.msi.umn.edu, you can use the "mb" command (make bucket):
s3cmd mb s3://mynewbucket
Note: There is a shared namespace for everyone. Replace mynewbucket with a bucket name of your choosing. Bucket names must follow these naming rules:
The bucket name can be between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes.
Each label in the bucket name must start with a lowercase letter or number.
The bucket name cannot contain underscores, end with a dash, have consecutive periods, or use dashes adjacent to periods.
The bucket name cannot be formatted as an IP address (198.51.100.24)
To copy a file to S3:
s3cmd put ~/localfile.txt s3://mynewbucket
To copy a file from S3 to your home directory:
s3cmd get s3://mynewbucket/localfile.txt ~/
To recursively copy a set of directories, use the 's3cmd sync' command.
s3cmd sync ~/mydirectory s3://mynewbucket
To recursively make a bucket and all its contents publically readable by everyone:
s3cmd setacl -P -r s3://mynewbucket
Warning - the above example will make all of your bucket contents available to anyone, over https as well as using s3cmd.
For more refined control over sharing content from tier2, please also see our documentation on bucket policies