True Cloud

Overview

When people talk of "the cloud" what they are really talking about is storage and computing resources rented from one of a small number of vary large companies who allocate them dynamically from within their large, power-hungry and not terribly popular data centres.

There are two groups of customers for this service - corporate customers who pay large fees and get contracts with teeth and private customers who don't. Both have to put a great deal of trust in the operators of their chosen cloud.

Nearly all of the private customers have computers that run well over ninety percent idle, have substantial free disc space and are well connected to the internet (domestic bandwidth keeps going up). These resources are widely distributed both geographically and by network topography.

Why not use the internet in a way that matches the original goal of bomb proof (literally) reliablity. Why not also make it open and free - or at least paid in kind. The idaa goes like this

• The user provides resources to the cloud from a permanently running and connected system such as a router or NAS.
• In return the user gets proportionate access to resources provided by other users including
  • Cloud storage: Redundant, encrypted, distributed with read-only snapshots
  • Cloud applications: Run containerised services on remote systems
  • A CDN running across the cloud: Publish with or without access controls
• The cloud is mediated by the resource management services (RMS) running in containers on the host systems (NAS/router) and interconnected over a web of VPNs. When the RMS comes up it queries a discovery service for management VPNs and joins between one and four - note that joining involves negotiation, the new node must fit within the toplogical and geographic distribution requirements for VPNs.
• The user's cloud storage is provided out of volumes from a number of remote and widely distributed systems arranged to provide three volumes of redundancy - so that the data is available provided even with three unavailable hosts. These volumes contain only encrypted data, the only place the keys to decrypt it exist is in a container running on the user's own system which stitches those volumes together into a ZFS pool backed by encrypted storage. All access to unencrypted data is via that container which only listens for instructions locally.
• Separate containers provide services giving access to data contained within this ZFS pool. Their own access is via NFS over a dedicated VPN, usually only to a specific subset of the user's cloud data (read only when possible). This is only controllable locally by the cloud filesystem UI.
• The overarching principle is resource sharing, provide local resources to the cloud and receive access to cloud resources in proportion. So for example in order to be able to use remote systems to run applications a user must allocate resources for others to run containers on their system,

Details

There are several parts to this.

• Resource management service
• Management VPN discovery service
• Volume provider
• Cloud file system
• Component Library
• CDN
• Hosting

Resource Management Service

This needs writing and maintaining as open source with open protocols. It shouldn't be too big a project once the core is designed, documented and built - THAT ORDER!

The RMS is a distributed service running in containers (such as Linux docker images or FreeBSD iocage jails) on each node. It is responsible for connecting the node to the cloud, sending resource requests, responding to incoming resource requests, managing local resources, participating in VPN management and keeping track of all of this so as to enforce cloud policies around fair use, resource sharing, VPN size and constitution.

The RMS will first connect to between one and four management VPNs using the disovery service that runs in the cloud. There will be many VPNs available interconnected to provide multiple fallback routes. Requests for resources will be broadast on each connected VPN.

All traffic across the VPNs is end-to-end encrypted except for broadcast resource requests.

The RMS will calculate the user's cloud storage entitlement based on their contribution (probably between one third and a half allowing for system overheads and outages) and set up a Cloud File System (CFS) container providing the entitlement as a parameter.

The RMS will calculate the user's processing entitlement based on their contribution. By default application containers run locally, if entitlement and confguration permit they may overflow to using remote resources to run the container. If the container requires access to some of the cloud data then a suitable export and dedicated VPN must be prepared in the CFS. Remote containers can also be explicitly requested by the user, subject to entitlement.

Common cloud services such as the CDN, the VPN discovery mechanism and the service library will be provided by containers running in resources provided by users. They will scale with the cloud based on a simple policy of requiring a suitable density in each VPN and implemented in the RMS.

RMS's need a mechanism to validate each other.

Management VPN Discovery Service

This has to be accessible from outside the cloud on the public internet. It must have a well known DNS entry and be secured by SSL.

From inside the cloud this service has to be backed by data stored on high reliability storage - mirrored volumes would seem appropriate with a lot of copies (the volume is low, the update rate is low, availability is important).

All that is stored and made available is a list of VPNs soliciting for content. For each VPN this is from inception until it is full.

Each VPN record consists of UUID, list of public IPs, guest port, guest key.

This data will be presented by a server running in a container somewhere in the cloud providing seed data to the cloud's CMS. The server will supply randomised selections of VPN records on demand as an RSS feed.

To connect to a VPN a node must first connect to the related guest VPN, establish its validity with the RMS there and provide the UUIDs for its geographical location and network endpoint. The gaurdian RMS will then evaluate whether this new node is a fit for the VPN - if so the connection details (including a freshly minted key) will be provided to the node's RMS. Only one guest VPN connection is permitted at a time.

Volume Provider

ANother containerised service that runs on each node. The VP manages the storage provided by the user. This storage gets used in a number of ways:

• Providing Volumes to back cloud filesystems for other users. This is the principle use.
• Providing local storage for cloud service containers. For example the data storage for a web server or working storage of some kind.
• Providing storage for cloud infrastructure such as the VPN discovery service and the CDN.

When the VP provides storage for non-local use that storage is provided over a dedicated VPN (nested in the management VPN) connected only to the client and using the iSCSI protocol.

The request protocol for a remote volume goes as follows:

• ->: requestee address, size
• <-: N/A or OK + geographical and network UUIDs
• ->: Y/N
• <-: N/A or VPN endpoint details, key and volume UUID

Cloud File System

This containerised service runs in a container that has only local connectivity initially.

On startup the CFS requestes the RMS to obtain eight volumes from eight well distributed VPs, each one fifth the size of the user's entitlement. Once these have been obtained it connects to the VPNs and stitches the volumes together into a ZFS RAIDZ3 pool.

The CFS exposes a UI to the host only providing:

• Manage filesystems in the pool - including encryption
• Manage snapshots for each filesystem
• Export filesystems to the host - r/w or r/o
• Manage dedicated VPN based safe external exports - r/w or r/o
• Manage replication to other pools - with dedicated VPN support

Component Library

This will contain containerised applications preferably in both docker and iocage formats. It may also contain some VM images, such as a VM set up to run the entire system ready to drop into any handy hypervisor. The library will of course be hosted in this cloud, and served using the CDN built into the management service.

The component library will have to make installer images and the VM image for container runners available over the public internet. These packages must be signed (preferably countersigned too) by properly airgapped signing systems. Everything else can be distributed only across cloud management VPNs.

The installer image should only be capable of connecting to a VPN as a guest and pulling a copy of the RMS to bootstrap the setup process. That way everything that is used to run the cloud comes from the cloud over a cloud VPN.

The RMS pulls the remainder of the component library - all of the other core components and user tools over the cloud management VPNs.

The library itself is exported from cloud storage belonging to the development team.

CDN

Running in containers on all nodes and using contributed storage, this is a pretty typical CDN (probably built with off the shelf components) with one twist.

It must be possible to seed material for publishing to the public internet and to seed material that is only accessible over cloud management VPNs.

Hosting

To participate in this cloud a user has to provide a machine with stable connectivity and a high uptime capable of running containers. One very natural implementation would be as a component that could be added to a NAS such as TrueNAS core, a router such as OpenWRT or on an SBC such as a Raspberry-Pi.

© 2023 Steve O'Hara-Smith