Class S3BlobStore
- java.lang.Object
-
- org.nuxeo.ecm.core.blob.AbstractBlobStore
-
- org.nuxeo.ecm.blob.s3.S3BlobStore
-
- All Implemented Interfaces:
BlobStore
public class S3BlobStore extends AbstractBlobStore
Blob storage in S3.- Since:
- 11.1
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description class
S3BlobStore.S3BlobGarbageCollector
Garbage collector for S3 binaries that stores the marked (in use) binaries in memory.-
Nested classes/interfaces inherited from interface org.nuxeo.ecm.core.blob.BlobStore
BlobStore.OptionalOrUnknown<T>
-
-
Field Summary
Fields Modifier and Type Field Description protected boolean
allowByteRange
protected com.amazonaws.services.s3.AmazonS3
amazonS3
protected String
bucketName
protected String
bucketPrefix
protected S3BlobStoreConfiguration
config
protected BinaryGarbageCollector
gc
protected static String
USER_METADATA_USERNAME
protected boolean
useVersion
If true, include the object version in the key.-
Fields inherited from class org.nuxeo.ecm.core.blob.AbstractBlobStore
BYTE_RANGE_SEP, keyStrategy, name
-
-
Constructor Summary
Constructors Constructor Description S3BlobStore(String name, S3BlobStoreConfiguration config, KeyStrategy keyStrategy)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
checkDigest(String expectedDigest, com.amazonaws.services.s3.transfer.Download download, Path file)
protected void
clearBucket()
protected boolean
copyBlob(String key, S3BlobStore sourceBlobStore, String sourceKey, boolean move)
boolean
copyBlob(String key, BlobStore sourceStore, String sourceKey, boolean atomicMove)
Writes a file based on a key, as a copy/move from a source in another blob store.protected void
copyBlob(S3BlobStoreConfiguration sourceConfig, String sourceKey, S3BlobStoreConfiguration destinationConfig, String destinationKey, boolean move)
protected String
copyBlob(Blob blob)
Copies the blob as a direct S3 operation, if possible.protected boolean
copyBlobGeneric(String key, BlobStore sourceStore, String sourceKey, boolean atomicMove)
boolean
copyBlobIsOptimized(BlobStore sourceStore)
Checks if blob copy/move from another blob store to this one can be done efficiently.void
deleteBlob(String key)
Deletes a blob based on a key.protected boolean
exists(String bucketKey)
BinaryGarbageCollector
getBinaryGarbageCollector()
Returns the binary garbage collector (GC).BlobStore.OptionalOrUnknown<Path>
getFile(String key)
Gets an already-existing file containing the blob for the given key, if present.S3BlobStore
getS3BinaryManager()
BlobStore.OptionalOrUnknown<InputStream>
getStream(String key)
Gets the stream of the blob for the given key, if present.boolean
hasVersioning()
Whether this blob store has versioning.protected boolean
isBucketVersioningEnabled()
protected static boolean
isMissingKey(com.amazonaws.AmazonServiceException e)
protected static boolean
isNotImplemented(com.amazonaws.AmazonServiceException e)
boolean
readBlob(String key, Path dest)
Reads a blob based on its key into the given file.protected void
setMetadata(com.amazonaws.services.s3.model.ObjectMetadata objectMetadata, BlobContext blobContext)
String
writeBlob(BlobWriteContext blobWriteContext)
Writes a blob.void
writeBlobProperties(BlobUpdateContext blobUpdateContext)
Sets properties on a blob.protected String
writeFile(String key, Path file, BlobContext blobContext, String fileTraceSource)
Writes a file with the given key and returns its version id.-
Methods inherited from class org.nuxeo.ecm.core.blob.AbstractBlobStore
deleteBlob, getByteRangeFromKey, getKeyStrategy, getName, logTrace, logTrace, logTrace, randomLong, randomString, setByteRangeInKey, stripBlobKeyPrefix, transfer, transfer, transfer, unwrap, writeBlob
-
-
-
-
Field Detail
-
USER_METADATA_USERNAME
protected static final String USER_METADATA_USERNAME
- See Also:
- Constant Field Values
-
config
protected final S3BlobStoreConfiguration config
-
amazonS3
protected final com.amazonaws.services.s3.AmazonS3 amazonS3
-
bucketName
protected final String bucketName
-
bucketPrefix
protected final String bucketPrefix
-
allowByteRange
protected final boolean allowByteRange
-
useVersion
protected final boolean useVersion
If true, include the object version in the key.
-
gc
protected final BinaryGarbageCollector gc
-
-
Constructor Detail
-
S3BlobStore
public S3BlobStore(String name, S3BlobStoreConfiguration config, KeyStrategy keyStrategy)
-
-
Method Detail
-
getS3BinaryManager
public S3BlobStore getS3BinaryManager()
-
isMissingKey
protected static boolean isMissingKey(com.amazonaws.AmazonServiceException e)
-
isNotImplemented
protected static boolean isNotImplemented(com.amazonaws.AmazonServiceException e)
-
isBucketVersioningEnabled
protected boolean isBucketVersioningEnabled()
-
hasVersioning
public boolean hasVersioning()
Description copied from interface:BlobStore
Whether this blob store has versioning.With versioning, two writes may use the same key. The returned keys will include a different version number to distinguish the writes.
- Specified by:
hasVersioning
in interfaceBlobStore
- Overrides:
hasVersioning
in classAbstractBlobStore
-
writeBlob
public String writeBlob(BlobWriteContext blobWriteContext) throws IOException
Description copied from interface:BlobStore
Writes a blob.Note that the returned key may be different than the one requested by the
BlobWriteContext
, if the blob store needs additional version info to retrieve it.- Parameters:
blobWriteContext
- the context of the blob write, including the blob- Returns:
- the key to use to read this blob in the future
- Throws:
IOException
-
writeFile
protected String writeFile(String key, Path file, BlobContext blobContext, String fileTraceSource) throws IOException
Writes a file with the given key and returns its version id.- Throws:
IOException
-
setMetadata
protected void setMetadata(com.amazonaws.services.s3.model.ObjectMetadata objectMetadata, BlobContext blobContext)
-
getFile
public BlobStore.OptionalOrUnknown<Path> getFile(String key)
Description copied from interface:BlobStore
Gets an already-existing file containing the blob for the given key, if present.Note that this method is best-effort, it may return unknown even though the blob exists in the store, it's just that it's not handily available locally in a file.
- Parameters:
key
- the blob key- Returns:
- the file containing the blob, or empty if the blob cannot be found, or unknown if no file is available locally
-
getStream
public BlobStore.OptionalOrUnknown<InputStream> getStream(String key) throws IOException
Description copied from interface:BlobStore
Gets the stream of the blob for the given key, if present.Note that this method is best-effort, it may return unknown even though the blob exists in the store, it's just that it's not efficient to return it as a stream.
- Parameters:
key
- the blob key- Returns:
- the blob stream, or empty if the blob cannot be found, or unknown if no stream is efficiently available
- Throws:
IOException
-
clearBucket
protected void clearBucket()
-
readBlob
public boolean readBlob(String key, Path dest) throws IOException
Description copied from interface:BlobStore
Reads a blob based on its key into the given file.- Parameters:
key
- the blob keydest
- the file to use to store the fetched data- Returns:
true
if the file was fetched,false
if the file was not found- Throws:
IOException
-
checkDigest
protected void checkDigest(String expectedDigest, com.amazonaws.services.s3.transfer.Download download, Path file) throws IOException
- Throws:
IOException
-
copyBlob
protected String copyBlob(Blob blob) throws IOException
Copies the blob as a direct S3 operation, if possible.- Returns:
- the key, or
null
if no copy was done - Throws:
IOException
-
copyBlobIsOptimized
public boolean copyBlobIsOptimized(BlobStore sourceStore)
Description copied from interface:BlobStore
Checks if blob copy/move from another blob store to this one can be done efficiently.- Specified by:
copyBlobIsOptimized
in interfaceBlobStore
- Overrides:
copyBlobIsOptimized
in classAbstractBlobStore
- Parameters:
sourceStore
- the source store- Returns:
true
if the copy/move can be done efficiently
-
copyBlob
public boolean copyBlob(String key, BlobStore sourceStore, String sourceKey, boolean atomicMove) throws IOException
Description copied from interface:BlobStore
Writes a file based on a key, as a copy/move from a source in another blob store.If the copy/move is requested to be atomic, then the destination file is created atomically. In case of atomic move, in some stores the destination will be created atomically but the source will only be deleted afterwards.
- Parameters:
key
- the keysourceStore
- the source storesourceKey
- the source keyatomicMove
-true
for an atomic move,false
for a regular copy- Returns:
true
if the file was found in the source store,false
if it was not found- Throws:
IOException
-
copyBlob
protected boolean copyBlob(String key, S3BlobStore sourceBlobStore, String sourceKey, boolean move) throws com.amazonaws.AmazonServiceException
- Returns:
false
if generic copy is needed- Throws:
com.amazonaws.AmazonServiceException
- if the source is missing
-
copyBlob
protected void copyBlob(S3BlobStoreConfiguration sourceConfig, String sourceKey, S3BlobStoreConfiguration destinationConfig, String destinationKey, boolean move)
-
copyBlobGeneric
protected boolean copyBlobGeneric(String key, BlobStore sourceStore, String sourceKey, boolean atomicMove) throws IOException
- Throws:
IOException
-
writeBlobProperties
public void writeBlobProperties(BlobUpdateContext blobUpdateContext) throws IOException
Description copied from interface:BlobStore
Sets properties on a blob.- Specified by:
writeBlobProperties
in interfaceBlobStore
- Overrides:
writeBlobProperties
in classAbstractBlobStore
- Parameters:
blobUpdateContext
- the blob update context- Throws:
IOException
-
deleteBlob
public void deleteBlob(String key)
Description copied from interface:BlobStore
Deletes a blob based on a key. No error occurs if the blob does not exist.This method does not throw
IOException
, but may log an error message.- Parameters:
key
- the blob key
-
getBinaryGarbageCollector
public BinaryGarbageCollector getBinaryGarbageCollector()
Description copied from interface:BlobStore
Returns the binary garbage collector (GC).Several calls to this method will return the same GC, so that its status can be monitored using
BinaryGarbageCollector.isInProgress()
.- Returns:
- the binary GC
-
-