S3¶

The S3 sandbox type can be used to access Amazon S3 buckets. The S3 sandbox is presented to the user as any other sandbox would, and thus is transparent.

Configuration Options¶

type¶

Type of the sandbox. For S3 type, set to S3.

path¶

Prefix where user should be sandboxed.

If path is not specified, bucket root is assumed. This is equivalent to specifying /.

accessKeyId¶

Access Key ID to use when accessing the bucket.

secretKey¶

Secret Key to use when accessing the bucket.

sessionToken¶

Session token used with temporary credentials.

bucketName¶

Bucket name to access. The bucket must already exist.

region¶

Region the bucket is in. Defaults to us-east-1.

Note

The specified region must match the bucket’s region. If not, the endpointOverride parameter must be used to point to the correct endpoint.

scheme¶

Scheme to use when accessing the endpoint. Must be one of https (default) or http. This is useful when specifying endpointOverride.

endpointOverride¶

Endpoint to use when accessing the bucket.

proxyScheme¶

Scheme to use when using a proxy server. Must be one of https or http (default). This is useful when specifying proxyHost.

proxyHost¶

Hostname of the proxy server.

proxyPort¶

Port for the proxy server.

proxyUserName¶

Username to authenticate against the proxy server.

proxyPassword¶

Password to authenticate against the proxy server.

verifySSL¶

Control if SSL connections should be verified. Use with caution.

caPath¶

Path to directory containing CA certificates.

caFile¶

Path to the file containing CA certificates. If neither caPath nor caFile is set, the server uses the system trust store.

useVirtualAdressing¶

Use virtual-hosted-style addressing. Defaults to true. Set to false when the endpoint needs path-style addressing.

Note

JetStream uses S3 version 2 API, including multipart uploads. When accessing third party endpoints, please ensure they are compatible with these requirements.

Note

JetStream confirms every write against the ETag that the store returns. A mismatch fails the file. The check is skipped for objects encrypted with SSE-KMS or SSE-C, because their ETag is not an MD5 of the content. To turn off this check, use jetstream server --disable-cloud-etag-verify.

Warning

In some cases, when transfers fail, JetStream can leave incomplete multipart uploads in the S3 bucket. By default, these uploads are not visible to users, and can take up space in the bucket. It is recommended that a bucket lifecycle rule is created for incomplete multipart uploads.

Memory Requirements¶

When storing data to S3 buckets, JetStream Server needs to do additional file caching. The size of file cache affects how many concurrent transfers can be stored in S3 at a time, as do the quality and conditions of the connection between the JetStream server and the S3 endpoint. The cache is not pre-allocated and will be allocated dynamically as necessary. Each JetStream transfer will reserve a small portion of the file cache for upload to or download from the S3 storage.

Size of the file cache can be adjusted using --max-cloud-cache-size option, and the per-transfer limits can be specified using the --max-cloud-cache-upload-size and --max-cloud-cache-download-size options.

For a larger cache, a cache file can be specified using --cloud-cache-file. If a cache file is specified, the server will use the file and not require additional memory. The given file should be on fast local storage, must be at least --max-cloud-cache-size bytes big, and should not be modified or deleting while the server is running.

Limitations¶

Files uploaded to S3 must be 5 TiB or smaller.
Objects named . and .. are ignored.
overwriteMode in createTransfer() is ignored. An upload always replaces an existing object with the same name.
If the connection drops, an upload to S3 resumes from the last part that JetStream gave to the store. If the receiving server restarts, the upload resumes from its last checkpoint when the store still holds the uploaded parts. Otherwise the upload restarts from the beginning.

Example¶

To sandbox all users into a bucket named sandboxBucket, you could use the following sandboxing configuration:

>>> api.server.setSandboxMapping('', {
    '': {
        'type': 'S3',
        'bucketName': 'sandboxBucket',
        'path': '/',
        'region': 'us-west-2',
        'accessKeyId': 'SECRETKEYID',
        'secretKey': 'secretkey'
    }
})