Second Tier Storage

How can my programs interface with Second Tier Storage?

Application Programming Interface (API) Support for S3 Access

For advanced tasks or in writing your own software, you may want to interact with Second Tier Storage directly through the S3 programming API. Libraries exist to do this from many programming languages.

  • boto is a useful Python library for accessing S3 gateways.  We have some simple examples at bottom of the page.
  • Amazon::S3 is a Perl library for working with and managing S3 buckets and keys.
  • AWS SDK for Java is developed directly by Amazon.

A full list of language libraries supported by Ceph S3 with many examples can be found in the Ceph documentation.

In every case, your program will need to use your s3 credentials to access the S3 service.

Supercomputing FAQs

Example: Programmatic Access to S3 credentials

To assist with obtaining one's s3 credentials programmatically, MSI has created an "s3info" command that retrieves the S3 credentials associated to the user that runs it.  This command is available on login nodes and on Agate. A --keys option causes it to provide easily parseable output, so you can run it from a bash script or from the language of your choice. The below shell session shows its usage with :

juser@login02 [~]% s3info
Access key: QPBEF6DD3AUY97V89DCH
Secret key: Zcl6xbjQSloQRp2tEdkUdZjPtOdJvN4hWgUH0j6h
Sharing address: [email protected]
juser@login02 [~]% s3info --keys
QPBEF6DD3AUY97V89DCH Zcl6xbjQSloQRp2tEdkUdZjPtOdJvN4hWgUH0j6h
juser@login02 [~]%

Example: Programmatic Access with Python

s3.msi.umn.edu can also be accessed programatically. boto is a useful Python library to interact with an S3 gateway. Below is an example of how to establish a connection and print a list of your buckets.

import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key

access_key = 'ABCDEFGHIJKLMNOP'
secret_key = 'kjsdlhfguihkejrg4ghih8a43934h2oih'
conn = S3Connection(access_key, secret_key, host='s3.msi.umn.edu')

rs = conn.get_all_buckets()
for b in rs:
    print b.name 

Create a URL that allows anyone with the URL to download a file in the next hour:

import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key

access_key = 'ABCDEFGHIJKLMNOP'
secret_key = 'kjsdlhfguihkejrg4ghih8a43934h2oih'
conn = S3Connection(access_key, secret_key, host='s3.msi.umn.edu')

bucket = conn.get_bucket('mybucket')
filekey = bucket.get_key('bigfiletoshare.txt')
url_to_share = filekey.generate_url(3600, query_auth=True, force_http=True)
print url_to_share 

Discover Advanced Computing and Data Solutions at MSI

Our Services