File Stores
A file store is responsible for:
- reading and writing files (.mp3, .pdf, etc.) to/from an object store.
- providing "transactions" that make all file operations atomic in a single request.
- declaring the native object store client it delegates persistence operations to.
If a file store implementation is unable to handle a file InputStream, Athena pushes these responsibilities to the object store.
Included Stores
Athena comes bundled with a number of file stores:
- Swift Store - A file store that can map operations on a file to an underlying OpenStack Swift API. Athena has explicit support for Swift
- HDFS Store - File is persisted on Hadoop HDFS.
It is assumed that the "HDFS Store" means a single-cluster HDFS. However, the Athena architecture does not preclude implementing a multi-cluster HDFS store
Stores can be included through the following artifact dependencies:
Swift Store
<dependency>
<groupId>io.github.qubitpi.athena</groupId>
<artifactId>athena-filestore-swift</artifactId>
<version>${version.athena}</version>
</dependency>
HDFS Store
<dependency>
<groupId>io.github.qubitpi.athena</groupId>
<artifactId>athena-filestore-hdfs</artifactId>
<version>${version.athena}</version>
</dependency>
Overriding the Store
To change the store, override the FileStore
binding. For example, to use a store called SomeCustomFileStore
:
public class AppBinderFactory extends AbstractBinderFactory {
...
@Override
protected Class<? extends FileStore> buildFileStore() {
return SomeCustomFileStore.class;
}
}
Custom Stores
Custom stores can be written by implementing the FileStore
interface. Take Amazon S3 for instance
@Singleton
public class S3FileStore implements FileStore {
private final AmazonS3 s3client;
private final FileIdGenerator fileIdGenerator;
@Inject
public S3FileStore(@NotNull final AmazonS3 s3client, @NotNull final FileIdGenerator fileIdGenerator) {
this.s3client = Objects.requireNonNull(s3client);
this.fileIdGenerator = Objects.requireNonNull(fileIdGenerator);
}
@Override
public String upload(final File file) {
final String fileId = fileIdGenerator.apply(file);
s3client.putObject(
file.getMetaData().getFileType().name(),
fileId,
file.getFileContent(),
new ObjectMetadata()
);
return fileId;
}
@Override
public InputStream download(final String fileId) {
return getS3client().getObject(...);
}
}
Multiple Stores
A common pattern in Athena is the need to support multiple file stores. Typically, one file store manages most files, but some others may require a different object storage backend or have other needs to specialize the behavior of the store. The multiplex store in Athena manages multiple stores - delegating calls to the appropriate store which is responsible for a particular file.
This is a feature yet to be offered soon.