S3¶

The S3 sandbox type can be used to access Amazon S3 buckets. The S3 sandbox is presented to the user as any other sandbox would, and thus is transparent.

Configuration Options¶

type¶

Type of the sandbox. For S3 type, set to S3.

path¶

Prefix where user should be sandboxed.

If path is not specified, bucket root is assumed. This is equivalent to specifying /.

accessKeyId¶

Access Key ID to use when accessing the bucket.

secretKey¶

Secret Key to use when accessing the bucket.

sessionToken¶

Session token used with temporary credentials.

bucketName¶

Bucket name to access. The bucket must already exist.

region¶

Region the bucket is in. Defaults to us-east-1.

Note

The specified region must match the bucket’s region. If not, the endpointOverride parameter must be used to point to the correct endpoint.

scheme¶

Scheme to use when accessing the endpoint. Must be one of https (default) or http. This is useful when specifying endpointOverride.

endpointOverride¶

Endpoint to use when accessing the bucket.

proxyScheme¶

Scheme to use when using a proxy server. Must be one of https or http (default). This is useful when specifying proxyHost.

proxyHost¶

Hostname of the proxy server.

proxyPort¶

Port for the proxy server.

proxyUserName¶

Username to authenticate against the proxy server.

proxyPassword¶

Password to authenticate against the proxy server.

verifySSL¶

Control if SSL connections should be verified. Use with caution.

caPath¶

Path to directory containing CA certificates.

caFile¶

Path to the file containing CA certificates.

Note

JetStream uses S3 version 2 API, including multipart uploads. When accessing third party endpoints, please ensure they are compatible with these requirements.

Warning

In some cases, when transfers fail, JetStream can leave incomplete multipart uploads in the S3 bucket. By default, these uploads are not visible to users, and can take up space in the bucket. It is recommended that a bucket lifecycle rule is created for incomplete multipart uploads.

Memory Requirements¶

When storing data to S3 buckets, JetStream Server needs to do additional file caching. The size of file cache affects how many concurrent transfers can be stored in S3 at a time, as do the quality and conditions of the connection between the JetStream server and the S3 endpoint. The cache is not pre-allocated and will be allocated dynamically as necessary. Each JetStream transfer will reserve a small portion of the file cache for upload to or download from the S3 storage.

Size of the file cache can be adjusted using --max-cloud-cache-size option, and the per-transfer limits can be specified using the --max-cloud-cache-upload-size and --max-cloud-cache-download-size options.

For a larger cache, a cache file can be specified using --cloud-cache-file. If a cache file is specified, the server will use the file and not require additional memory. The given file should be on fast local storage, must be at least --max-cloud-cache-size bytes big, and should not be modified or deleting while the server is running.

Limitations¶

Files uploaded to S3 must be under one (1) Terabyte in size.
Objects named . and .. are ignored.
overwriteMode in createTransfer() is ignored, and OVERWRITE_IN_PLACE_ALWAYS is assumed.
Uploads to S3 do not support checkpoints. Interrupted transfers may restart from beginning.

Example¶

To sandbox all users into a bucket named sandboxBucket, you could use the following sandboxing configuration:

>>> api.server.setSandboxMapping('', {
    '': {
        'type': S3',
        'bucketName': 'sandboxBucket',
        'path': '/',
        'region': 'us-west-2',
        'accessKeyId': 'SECRETKEYID',
        'secretKey': 'secretkey'
    }
})