dans-layer-store-lib¶
A library for hierarchical storage of files in a layered way.
Layer store¶
A layer store is a hierarchical storage of files in a layered way. A layer contains a subset of the files and folders in the store. A layer is identified by a unique Layer ID. This ID is a Unix timestamp in milliseconds. The layer IDs determine the order in which the layers are stacked. The contents of a layer store can be transformed to a regular directory structure by stacking the layers in the order of their IDs. The layer with the highest ID is the top layer. Files in newer layers override files with the same path in older layers.
An example of a layer store with three layers is shown below.
+-------------------------------------------------------+
| Layer ID: 1706612162 |
| . |
| └── otherpath |
| └── to |
| └── otherdir |
| └── file3.txt |
+-------------------------------------------------------+
+-------------------------------------------------------+
| Layer ID: 1705633401 |
| . |
| └── path |
| └── to |
| └── dir1 |
| ├── file1.txt |
| └── file2.txt |
+-------------------------------------------------------+
+-------------------------------------------------------+
| Layer ID: 1701118823 |
| . |
| └── path |
| └── to |
| └── dir1 |
| └── file1.txt |
+-------------------------------------------------------+
Transforming this layer store to a regular directory structure results in the following directory structure:
+--------------------------------------------------------+
| ├── otherpath |
| │ └── to |
| │ └── otherdir |
| │ └── file3.txt |
| └── path |
| └── to |
| └── dir1 |
| ├── file1.txt |
| └── file2.txt |
+--------------------------------------------------------+
Note that:
file1.txtin layer1705633401overwritesfile1.txtin layer1701118823.- Adding files is done by adding them to the top layer.
- Removing files entails removing them from all layers (this is not shown in the example).
Layers are envisioned to be implemented as a set of archive files. A simple way to picture the transformation from layer store to directory structure is to imagine the archive files being extracted to one temporary directory, one after the other, in the order of their layer ID.
The ItemStore interface¶
The dans-layer-store-lib defines and implements an ItemStore interface. An ItemStore models basically a normal file/folder hierarchy. It provides methods
for adding, removing and retrieving items. In the scenario described above, it corresponds to the directory structure that results from transforming the layer
store to a regular directory structure.
The term item instead of file or folder is used to clearly distinguish between the model and the implementation, especially because the implementation will involve "physical" files and folders.
Although ItemStore does not expose the concept of a layer, the purpose of this library is to provide a layered storage. The ItemStore interface is intended
to hide the layering from the user. It is not really intended to be implemented in other ways, although that is possible. The interface is geared towards
providing exactly the features needed to implement ocfl-java's Storage interface without depending on any OCFL specific concepts. The two are connected
through the dans-ocfl-java-extensions-lib.
Layer database¶
The layer store is backed by a database. The purpose is to make operations that would otherwise be slow much faster. For example, listing the contents of a folder in a layer store with many layers would require reading all layers. Depending on the type of archive, it could also mean reading all archive files. If layer archives were stored in a very slow medium, such as tape, the problem would be compounded.
To solve this problem, the layer database stores a record for each item in each layer. The record contains the item's path and type. If it is a file, the record may also contain the entire content of the file. For which files the content is stored is configurable. Obviously, storing the content of all files would be very expensive in terms of storage space, so it is recommended to only store the content of files that are expected to be relatively small and need to be read often.
Layer states¶
A layer is always in one of three states:
- OPEN – this means it can be written to. It is the initial state of a layer. The files in the layer have been staged to fast storage.
- CLOSED – this means it cannot be written to. A layer must be closed before it can be archived. Closing a layer is a fast operation, but archiving can take a long time, depending on the size of the layer.
- ARCHIVED – this means the layer has been written to an archive file and the staged files have been deleted.
The transitions between these states are as follows:
- close (transition: OPEN -> CLOSED). This is an instantaneous operation. The files are still staged on fast storage after closing, but the layer can no longer be changed.
- archive (transition: CLOSED -> ARCHIVED). This is potentially a slow operation, as it involves writing the layer to an archive file and deleting the
staged files. The archive transition takes a boolean argument,
overwrite, which determines whether to overwrite an existing archive file if it exists. Ifoverwriteisfalse, an error is thrown if the archive file already exists. If it is true, the existing archive is only deleted after the new archive is fully written. - reopen (transition: ARCHIVED or CLOSED -> OPEN). This is potentially a slow operation, as it involves reading the archive file and staging the files to
fast storage again. The archive is not deleted after reopening. If the open layer is subsequently edited, closed and archived again, the client should
pass
overwrite= trueto make sure the updates are reflected in the archive file. It is also possible to reopen a CLOSED layer, which is a fast operation that does not involve reading the archive file.