- Provide loosely-coupled flexible distributed easy to use architecture
- Build on top of existing solutions
- Use data storage federation to address these problems.
- Present them via a standardized protocol that can be also mounted
- Provide a file system abstraction
- Introduce of a common management layer that loosely couples independent storage resources
- Distributed applications have a global shared view of the whole available storage space
- Applications can be developed locally and deployed on the cloud platform without changing the data access parameters
- Use storage space efficiently with the copy-on-write strategy
- Replication of data can be based on efficiency cost measures
- Reduce the risk of vendor lock-in in clouds since no large amount of data are on a single provider
- LOBCDER is a storage federation service making available distributed unstuctured data stored in various storage framework and independent providers
- LOBCDER loosely couples a variety of storage technologies such as Openstack-Swift ,iRODS GridFTP
- LOBCDER is a distributed file system that aims for “transparency” in a number of aspects
- It can be “invisible” to clients which “see” a system similar to a local file system
- Behind the scenes, it handles locating files, transporting data providing:
- Access transparency: clients are unaware that files are distributed and can access them in the same way as local files are accessed
- Location transparency: a consistent namespace encompasses remote files. The name of a file does not give its location
- Concurrency transparency: all clients have the same view of the state of the file system
- Heterogeneity: provided across different hardware operating system platforms
- Replication transparency: replicate files across multiple servers. Clients are unaware of this
- Migration transparency: files are able to move around without the client’s knowledge
It can also provide more advanced functionality to the rest of the modules in the VPH-Share cloud platform
System Overview (Fonrtend Layer)
- The frontend provides access control, authentication and authorization
- It is a WebDAV servlet which provides interoperability as an RFC standard
It enables network transparency through the use of numerous clients that are able to mount WebDAV
- It supports versioning, locking, and custom properties
- Authentication and authorization is delegated to the authentication service
- The authentication service authenticates user according to a security token
- The authentication service validates the token and returns information about the user
- For clients that want control over properties that depend on the infrastructure we have implemented a REST interface
System Overview (Resource Layer)
- The resource layer creates a logical representation of the physical storage space
and to manage the physical files
- The WebDAVResourceFactory, and the WebDAVResource provide a WebDAV representation of the LogicalResource
- The ResourceCatalog connects to the persistence layer and queries
- The Task component manages the physical files. It schedules file’s replication and delition
- The LogicalResources hold basic metadata such as modification date, length, etc.
- The PDRI component represents the physical data.
- The StorageSite component provides a description for the storage resources
- The backend layer provides the necessary abstraction to uniformly access physical storage resources.
- Virtual Resource System API:
- The VFSClient can perform file system operations on physical data.
- Different VFSDriver implementations allow transparent access to storage resources
- The persistence layer is a relational database which holds the logical data that are represented by the LogicalResource
- It provides Atomicity, Consistency, Isolation and Durability (ACID).
These properties are necessary in a multiuser environment for maintaining a synchronized and consistent view of the shared file system.
More details about this work can be found in:
- S. Koulouzis, D. Vasyunin, R.S. Cushing, A.S.Z. Belloum, Cloud Data Storage Federation for Scientific Applications, In Proceedings of the Euro-Par 2013: Parallel Processing Workshops, Lecture Notes in Computer Science, Aachen, Germany, Aug 2013.
- Cloud Federation for Sharing Scientific Data S. Koulouzis, R. Cushing, D. Vasunin, A.S.Z Belloum and M.T. Bubak 8th IEEE International Conference on eScience (eScience 2012) Chicago, Illinois, 8-12 October 2012. [poster]