Package org.nuxeo.ecm.core.storage.sql

Class S3BinaryManager

java.lang.Object

org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager

org.nuxeo.ecm.core.blob.binary.CachingBinaryManager

org.nuxeo.ecm.blob.AbstractCloudBinaryManager

org.nuxeo.ecm.core.storage.sql.S3BinaryManager

All Implemented Interfaces:

S3ManagedTransfer, BinaryManager, BlobProvider

Direct Known Subclasses:

CloudFrontBinaryManager

public class S3BinaryManager
extends AbstractCloudBinaryManager
implements S3ManagedTransfer

A Binary Manager that stores binaries as S3 BLOBs

The BLOBs are cached locally on first access for efficiency.

Because the BLOB length can be accessed independently of the binary stream, it is also cached in a simple text file if accessed before the stream.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	S3BinaryManager.S3BinaryGarbageCollector	Garbage collector for S3 binaries that stores the marked (in use) binaries in memory.
class	S3BinaryManager.S3FileStorage

Field Summary

Fields

Modifier and Type	Field	Description
static String	ACCELERATE_MODE_PROPERTY
protected com.amazonaws.services.s3.AmazonS3	amazonS3
static String	AWS_ID_PROPERTY
static String	AWS_SECRET_PROPERTY
static String	AWS_SESSION_TOKEN_PROPERTY
protected com.amazonaws.auth.AWSCredentialsProvider	awsCredentialsProvider
static String	BUCKET_NAME_PROPERTY
static String	BUCKET_PREFIX_PROPERTY
static String	BUCKET_REGION_PROPERTY
protected String	bucketName
protected String	bucketNamePrefix
protected com.amazonaws.ClientConfiguration	clientConfiguration
static String	CONNECTION_MAX_PROPERTY	AWS ClientConfiguration default 50
static String	CONNECTION_RETRY_PROPERTY	AWS ClientConfiguration default 3 (with exponential backoff)
static String	CONNECTION_TIMEOUT_PROPERTY	AWS ClientConfiguration default 50*1000 = 50s
protected com.amazonaws.services.s3.model.CryptoConfiguration	cryptoConfiguration
static String	DELIMITER
static String	DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT
static String	DIRECTDOWNLOAD_PROPERTY_COMPAT
protected com.amazonaws.services.s3.model.EncryptionMaterials	encryptionMaterials
static String	ENDPOINT_PROPERTY
protected boolean	isEncrypted
static String	KEYSTORE_FILE_PROPERTY
static String	KEYSTORE_PASS_PROPERTY
static String	PATHSTYLEACCESS_PROPERTY
static String	PRIVKEY_ALIAS_PROPERTY
static String	PRIVKEY_PASS_PROPERTY
static String	SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY
static String	SERVERSIDE_ENCRYPTION_PROPERTY
protected String	serverSideKMSKeyID
static String	SOCKET_TIMEOUT_PROPERTY	AWS ClientConfiguration default 50*1000 = 50s
static String	SYSTEM_PROPERTY_PREFIX
protected com.amazonaws.services.s3.transfer.TransferManager	transferManager
protected boolean	useServerSideEncryption

Fields inherited from class org.nuxeo.ecm.blob.AbstractCloudBinaryManager

CACHE_COUNT_PROPERTY, CACHE_MIN_AGE_PROPERTY, CACHE_SIZE_PROPERTY, DEFAULT_CACHE_COUNT, DEFAULT_CACHE_MIN_AGE, DEFAULT_CACHE_SIZE, DEFAULT_DIRECTDOWNLOAD, DEFAULT_DIRECTDOWNLOAD_EXPIRE, DIGEST_ALGORITHM_PROPERTY, directDownload, DIRECTDOWNLOAD_EXPIRE_PROPERTY, DIRECTDOWNLOAD_PROPERTY, directDownloadExpire, namespace, transientFlag

Fields inherited from class org.nuxeo.ecm.core.blob.binary.CachingBinaryManager

cachedir, fileCache, fileStorage

Fields inherited from class org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager

blobProviderId, DEFAULT_DEPTH, DEFAULT_DIGEST, descriptor, digestPattern, DIGESTS_BY_LENGTH, garbageCollector, MAX_BUF_SIZE, MD5_DIGEST, MD5_DIGEST_LENGTH, MIN_BUF_SIZE, properties, SHA1_DIGEST, SHA1_DIGEST_LENGTH, SHA256_DIGEST, SHA256_DIGEST_LENGTH

Fields inherited from interface org.nuxeo.ecm.core.blob.binary.BinaryManager

PROP_KEY, PROP_PATH

Constructor Summary

Constructors

Constructor	Description
S3BinaryManager()

Method Summary

Modifier and Type	Method	Description
protected void	abortOldMultipartUploadsInternal()
protected void	abortOldUploads()	Aborts uploads that crashed and are older than 1 day.
boolean	canAccessBucket()	Used in the healthCheck; the transferManager should be initialized and the bucket accessible
void	close()	Closes the binary manager and releases all resources and temporary objects held by it.
protected String	copyBlob(S3BinaryManager sourceBlobProvider, String blobKey)	Copies a blob.
com.amazonaws.services.s3.AmazonS3	getAmazonS3()	Gets AmazonS3.
com.amazonaws.auth.AWSCredentialsProvider	getAwsCredentialsProvider()	Gets the AWSCredentialsProvider.
String	getBucketName()	Gets the bucket name.
String	getBucketPrefix()	Gets the bucket prefix.
protected FileStorage	getFileStorage()
protected URI	getRemoteUri(String digest, ManagedBlob blob, javax.servlet.http.HttpServletRequest servletRequest)
protected String	getSystemPropertyPrefix()	Gets the prefix used for configuration using system properties.
com.amazonaws.services.s3.transfer.TransferManager	getTransferManager()	Returns the TransferManager.
protected BinaryGarbageCollector	instantiateGarbageCollector()
static boolean	isMD5(String digest)	Deprecated. since 11.1, now unused
protected static boolean	isMissingKey(com.amazonaws.AmazonClientException e)
void	removeBinaries(Collection<String> digests)	Remove definitively a set of binaries
protected void	removeBinary(String digest)
protected void	setupCloudClient()	Configure Cloud client using properties
String	writeBlob(Blob blob)	Writes a Blob to storage and returns information about it.

Methods inherited from class org.nuxeo.ecm.blob.AbstractCloudBinaryManager

getBinaryManager, getBooleanProperty, getContentDispositionHeader, getContentTypeHeader, getDefaultDigestAlgorithm, getIntProperty, getIntProperty, getProperties, getProperty, getProperty, getSystemPropertyName, getURI, initialize, isDirectDownload, isTransient, performsExternalAccessControl, readBlob, supportsUserUpdate, supportsUserUpdateDefaultTrue

Methods inherited from class org.nuxeo.ecm.core.blob.binary.CachingBinaryManager

getBinary, getBinary, getFile, initializeCache, initializeCache, initializeCache

Methods inherited from class org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager

computeDigestPattern, getBinary, getDescriptor, getDigestAlgorithm, getGarbageCollector, isValidDigest, setDescriptor, storeAndDigest, toHexString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nuxeo.ecm.core.blob.BlobProvider

allowByteRange, canConvert, deleteBlob, getAppLinks, getAvailableConversions, getBinaryGarbageCollector, getFile, getStatus, getStream, getStream, getThumbnail, hasCreateFromKeyPermission, isRecordMode, isTransactional, isVersion, readBlob, supportsSync, updateBlob, writeBlob

Field Detail

SYSTEM_PROPERTY_PREFIX

public static final String SYSTEM_PROPERTY_PREFIX

See Also:

Constant Field Values

BUCKET_NAME_PROPERTY

public static final String BUCKET_NAME_PROPERTY

See Also:

Constant Field Values

BUCKET_PREFIX_PROPERTY

public static final String BUCKET_PREFIX_PROPERTY

See Also:

Constant Field Values

BUCKET_REGION_PROPERTY

public static final String BUCKET_REGION_PROPERTY

See Also:

Constant Field Values

AWS_ID_PROPERTY

public static final String AWS_ID_PROPERTY

See Also:

Constant Field Values

AWS_SECRET_PROPERTY

public static final String AWS_SECRET_PROPERTY

See Also:

Constant Field Values

AWS_SESSION_TOKEN_PROPERTY

public static final String AWS_SESSION_TOKEN_PROPERTY

Since:

10.10

See Also:

Constant Field Values

CONNECTION_MAX_PROPERTY

public static final String CONNECTION_MAX_PROPERTY

AWS ClientConfiguration default 50

See Also:

Constant Field Values

CONNECTION_RETRY_PROPERTY

public static final String CONNECTION_RETRY_PROPERTY

AWS ClientConfiguration default 3 (with exponential backoff)

See Also:

Constant Field Values

CONNECTION_TIMEOUT_PROPERTY

public static final String CONNECTION_TIMEOUT_PROPERTY

AWS ClientConfiguration default 50*1000 = 50s

See Also:

Constant Field Values

SOCKET_TIMEOUT_PROPERTY

public static final String SOCKET_TIMEOUT_PROPERTY

AWS ClientConfiguration default 50*1000 = 50s

See Also:

Constant Field Values

KEYSTORE_FILE_PROPERTY

public static final String KEYSTORE_FILE_PROPERTY

See Also:

Constant Field Values

KEYSTORE_PASS_PROPERTY

public static final String KEYSTORE_PASS_PROPERTY

See Also:

Constant Field Values

SERVERSIDE_ENCRYPTION_PROPERTY

public static final String SERVERSIDE_ENCRYPTION_PROPERTY

See Also:

Constant Field Values

SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY

public static final String SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY

See Also:

Constant Field Values

PRIVKEY_ALIAS_PROPERTY

public static final String PRIVKEY_ALIAS_PROPERTY

See Also:

Constant Field Values

PRIVKEY_PASS_PROPERTY

public static final String PRIVKEY_PASS_PROPERTY

See Also:

Constant Field Values

ENDPOINT_PROPERTY

public static final String ENDPOINT_PROPERTY

See Also:

Constant Field Values

PATHSTYLEACCESS_PROPERTY

public static final String PATHSTYLEACCESS_PROPERTY

Since:

10.3

See Also:

Constant Field Values

ACCELERATE_MODE_PROPERTY

public static final String ACCELERATE_MODE_PROPERTY

Since:

11.1

See Also:

Constant Field Values

DIRECTDOWNLOAD_PROPERTY_COMPAT

public static final String DIRECTDOWNLOAD_PROPERTY_COMPAT

See Also:

Constant Field Values

DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT

public static final String DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT

See Also:

Constant Field Values

DELIMITER

public static final String DELIMITER

See Also:

Constant Field Values

bucketName

protected String bucketName

bucketNamePrefix

protected String bucketNamePrefix

awsCredentialsProvider

protected com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider

clientConfiguration

protected com.amazonaws.ClientConfiguration clientConfiguration

encryptionMaterials

protected com.amazonaws.services.s3.model.EncryptionMaterials encryptionMaterials

isEncrypted

protected boolean isEncrypted

cryptoConfiguration

protected com.amazonaws.services.s3.model.CryptoConfiguration cryptoConfiguration

useServerSideEncryption

protected boolean useServerSideEncryption

serverSideKMSKeyID

protected String serverSideKMSKeyID

amazonS3

protected com.amazonaws.services.s3.AmazonS3 amazonS3

transferManager

protected com.amazonaws.services.s3.transfer.TransferManager transferManager

Constructor Detail

S3BinaryManager

public S3BinaryManager()

Method Detail

close

public void close()

Description copied from interface: BinaryManager

Closes the binary manager and releases all resources and temporary objects held by it.

Specified by:

close in interface BinaryManager

Specified by:

close in interface BlobProvider

Overrides:

close in class CachingBinaryManager

abortOldUploads

protected void abortOldUploads()

Aborts uploads that crashed and are older than 1 day.

Since:

7.2

abortOldMultipartUploadsInternal

protected void abortOldMultipartUploadsInternal()

setupCloudClient

protected void setupCloudClient()
                         throws IOException

Description copied from class: AbstractCloudBinaryManager

Configure Cloud client using properties

Specified by:

setupCloudClient in class AbstractCloudBinaryManager

Throws:

IOException

getTransferManager

public com.amazonaws.services.s3.transfer.TransferManager getTransferManager()

Description copied from interface: S3ManagedTransfer

Returns the TransferManager.

Specified by:

getTransferManager in interface S3ManagedTransfer

removeBinary

protected void removeBinary(String digest)

getSystemPropertyPrefix

protected String getSystemPropertyPrefix()

Description copied from class: AbstractCloudBinaryManager

Gets the prefix used for configuration using system properties.

Specified by:

getSystemPropertyPrefix in class AbstractCloudBinaryManager

instantiateGarbageCollector

protected BinaryGarbageCollector instantiateGarbageCollector()

Specified by:

instantiateGarbageCollector in class AbstractCloudBinaryManager

removeBinaries

public void removeBinaries(Collection<String> digests)

Description copied from interface: BinaryManager

Remove definitively a set of binaries

Specified by:

removeBinaries in interface BinaryManager

Specified by:

removeBinaries in class AbstractCloudBinaryManager

Parameters:

digests - a set of digests, must not be null.

isMissingKey

protected static boolean isMissingKey(com.amazonaws.AmazonClientException e)

isMD5

@Deprecated
public static boolean isMD5(String digest)

Deprecated.

since 11.1, now unused

canAccessBucket

public boolean canAccessBucket()

Used in the healthCheck; the transferManager should be initialized and the bucket accessible

Since:

9.3

getFileStorage

protected FileStorage getFileStorage()

Specified by:

getFileStorage in class AbstractCloudBinaryManager

getAwsCredentialsProvider

public com.amazonaws.auth.AWSCredentialsProvider getAwsCredentialsProvider()

Gets the AWSCredentialsProvider.

Since:

10.2

getAmazonS3

public com.amazonaws.services.s3.AmazonS3 getAmazonS3()

Gets AmazonS3.

Since:

10.2

getBucketName

public String getBucketName()

Gets the bucket name.

Since:

11.1

getBucketPrefix

public String getBucketPrefix()

Gets the bucket prefix.

Since:

11.1

writeBlob

public String writeBlob(Blob blob)
                 throws IOException

Description copied from interface: BlobProvider

Writes a Blob to storage and returns information about it.

Called to store a user-created blob.

Specified by:

writeBlob in interface BlobProvider

Overrides:

writeBlob in class AbstractCloudBinaryManager

Parameters:

blob - the blob

Returns:

the blob key

Throws:

IOException

copyBlob

protected String copyBlob(S3BinaryManager sourceBlobProvider,
                          String blobKey)
                   throws IOException

Copies a blob. Returns null if the copy was not possible.

Parameters:

sourceBlobProvider - the source blob provider

blobKey - the source blob key

Returns:

the copied blob key, or null if the copy was not possible

Throws:

IOException

Since:

10.1

getRemoteUri

protected URI getRemoteUri(String digest,
                           ManagedBlob blob,
                           javax.servlet.http.HttpServletRequest servletRequest)
                    throws IOException

Overrides:

getRemoteUri in class AbstractCloudBinaryManager

Throws:

IOException