diff --git a/Docs/Abstractions.md.org b/Docs/Abstractions.md.org new file mode 100644 index 0000000..2254b73 --- /dev/null +++ b/Docs/Abstractions.md.org @@ -0,0 +1,35 @@ +* Abstractions +:PROPERTIES: +:CUSTOM_ID: abstractions +:END: +| [[file:Installation.md][◀ Previous]] | [[file:Implementation.md][Next ▶]] | +|--------------------------------------+------------------------------------| + +The Abstractions package consists of black-boxed functions for P2PRC. + +** Functions +:PROPERTIES: +:CUSTOM_ID: functions +:END: +- =Init()=: Initializes P2PRC with all the needed + configurations. +- =Start()=: Starts p2prc as a server and makes it possible to extend by + adding other routes and functionality to P2PRC. +- =MapPort()=: On the local machine the port you want to export + to world. +- =StartContainer()=: The machine on the p2p network where + you want to spin up a docker container. +- =RemoveContainer(,)=: Terminate container + based on the IP address and container name. +- =GetSpecs()=: Get specs of a machine on the network based + on the IP address. +- =ViewIPTable()=: View the IP table which about nodes in the network. +- =UpdateIPTable()=: Force update IP table to learn about new nodes + faster. + +-------------- + +*** Next Chapter: [[file:Implementation.md][Implementation]] +:PROPERTIES: +:CUSTOM_ID: next-chapter-implementation +:END: diff --git a/Docs/Bindings.md.org b/Docs/Bindings.md.org new file mode 100644 index 0000000..fa77c64 --- /dev/null +++ b/Docs/Bindings.md.org @@ -0,0 +1,180 @@ +* Language Bindings +:PROPERTIES: +:CUSTOM_ID: language-bindings +:END: +[[https://en.wikipedia.org/wiki/Language_binding][Language bindings]] +refers to wrappers to bridge 2 programming languages. This is used in +P2PRC to extend calling P2PRC functions in other programming languages. +Currently this is done by generating =.so= and =.h= from the Go +compiler. + +** How to build shared object files +:PROPERTIES: +:CUSTOM_ID: how-to-build-shared-object-files +:END: +**** The easier way +:PROPERTIES: +:CUSTOM_ID: the-easier-way +:END: +#+begin_src sh +# Run +make sharedObjects +#+end_src + +**** Or the direct way +:PROPERTIES: +:CUSTOM_ID: or-the-direct-way +:END: +#+begin_src sh +# Run +cd Bindings && go build -buildmode=c-shared -o p2prc.so +#+end_src + +**** If successfully built: +:PROPERTIES: +:CUSTOM_ID: if-successfully-built +:END: +#+begin_src sh +# Enter into the Bindings directory +cd Bindings +# List files +ls +# Find files +p2prc.h p2prc.so +#+end_src + +** Workings under the hood +:PROPERTIES: +:CUSTOM_ID: workings-under-the-hood +:END: +Below are a sample set of commands to open the bindings implementation. + +#+begin_example +# run +cd Bindings/ +# list files +ls +# search for file +Client.go +#+end_example + +*** In Client go +:PROPERTIES: +:CUSTOM_ID: in-client-go +:END: +There a few things to notice which are different from your standard Go +programs: + +**** 1. We import "C" which means [[https://pkg.go.dev/cmd/cgo][Cgo]] is required. +:PROPERTIES: +:CUSTOM_ID: we-import-c-which-means-cgo-is-required. +:END: +#+begin_src go +import "C" +#+end_src + +**** 2. All functions which are required to be called from other programming languages have comment such as. +:PROPERTIES: +:CUSTOM_ID: all-functions-which-are-required-to-be-called-from-other-programming-languages-have-comment-such-as. +:END: +#+begin_src go +//export + +// ------------ Example ---------------- +// The function below allows to externally +// to call the P2PRC function to start containers +// in a specific node in the know list of nodes +// in the p2p network. +// Note: the comment "//export StartContainer". + +//export StartContainer +func StartContainer(IP string) (output *C.char) { + container, err := client.StartContainer(IP, 0, false, "", "") + if err != nil { + return C.CString(err.Error()) + } + return ConvertStructToJSONString(container) + } +#+end_src + +**** 3. While looking through the file (If 2 files are compared it is pretty trivial to notice a common structure). +:PROPERTIES: +:CUSTOM_ID: while-looking-through-the-file-if-2-files-are-compared-it-is-pretty-trivial-to-notice-a-common-structure. +:END: +#+begin_src go +// --------- Example ------------ + +//export StartContainer +func StartContainer(IP string) (output *C.char) { + container, err := client.StartContainer(IP, 0, false, "", "") + if err != nil { + return C.CString(err.Error()) + } + return ConvertStructToJSONString(container) +} + +//export ViewPlugin +func ViewPlugin() (output *C.char) { + plugins, err := plugin.DetectPlugins() + if err != nil { + return C.CString(err.Error()) + } + return ConvertStructToJSONString(plugins) +} +#+end_src + +**** It is easy to notice that: +:PROPERTIES: +:CUSTOM_ID: it-is-easy-to-notice-that +:END: +- =ConvertStructToJSONString()=: This is a helper function + that convert a go object to JSON string initially and converts it to + =CString=. +- =(output *C.char)=: This is the return type for most of the functions. + +**** A Pseudo code to refer to the common function implementation shape could be represented as: +:PROPERTIES: +:CUSTOM_ID: a-pseudo-code-to-refer-to-the-common-function-implementation-shape-could-be-represented-as +:END: +#+begin_example +func (output *C.char) { + , := () + if != nil { + return C.CString(.Error()) + } + return ConvertStructToJSONString() +} +#+end_example + +** Current languages supported +:PROPERTIES: +:CUSTOM_ID: current-languages-supported +:END: +- Python + +*** Build sample python program +:PROPERTIES: +:CUSTOM_ID: build-sample-python-program +:END: +The easier way + +#+begin_src sh +# Run +make python +# Expected ouput +Output is in the Directory Bindings/python/export/ +# Run +cd Bindings/python/export/ +# list files +ls +# Expected output +SharedObjects/ p2prc.py +#+end_src + +Above shows a generated folder which consists of a folder called +"SharedObjects/" which consists of =p2prc.so= and =p2prc.h= files. +=p2prc.py= refers to a sample python script calling P2PRC go functions. +To start an any project to extend P2PRC with python, This generated +folder can copied and created as a new git repo for P2PRC extensions +scripted or used a reference point as proof of concept that P2PRC can be +called from other programming languages. diff --git a/Docs/CliImplementation.md.org b/Docs/CliImplementation.md.org new file mode 100644 index 0000000..c9940c4 --- /dev/null +++ b/Docs/CliImplementation.md.org @@ -0,0 +1,24 @@ +* Cli module +:PROPERTIES: +:CUSTOM_ID: cli-module +:END: +The Cli (i.e Command Line Interface) is the only one in which the user +can directly interact with the modules in the project. The objective +when building the Cli was to have the least amount of commands as +possible. The cli was built using the library called urfave cli v2 . +They were 2 major files created named as flags.go and actions.go. ### +Flags.go The flags .go file is responsible to create the appropriate +flags for the cli. There are 2 types of flags called boolean and string. +Each of the flags outputs are assigned to a variable to be handled. The +flags can also detect environment variables set. This feature is useful +because if the user wants to call certain flags in a repeated sequence +it only has to be initialized once. + +*** Actions.go +:PROPERTIES: +:CUSTOM_ID: actions.go +:END: +The actions.go file is implemented to call the appropriate functions +when the flags are called. It interacts directly with the modules in the +project. Action.go checks if variables are not empty string or the +boolean value is true. diff --git a/Docs/Client.md.org b/Docs/Client.md.org new file mode 100644 index 0000000..0142853 --- /dev/null +++ b/Docs/Client.md.org @@ -0,0 +1,33 @@ +* Client Module +:PROPERTIES: +:CUSTOM_ID: client-module +:END: +This module is incharge of communicating with the server and receiving +the appropriate information back from the server. + +** Functions of the Client Module +:PROPERTIES: +:CUSTOM_ID: functions-of-the-client-module +:END: + +#+begin_html + +#+end_html + +- [[#decision-maker-on-how-the-ip-table-is-created-or-updated][Decision + maker on how the ip table is created or updated]] + +** Decision maker on how the IP table is created or updated +:PROPERTIES: +:CUSTOM_ID: decision-maker-on-how-the-ip-table-is-created-or-updated +:END: +- Does a local speedtest to verify and see if the server IP's in the IP + table are pingable. +- Tries to ping the servers IP Table addresses. +- If it's pingable then it's added as a new entry in the IP table. +- The following steps occurs in the clients IP table. +- To ensure that the same servers are not being called to update the IP + table. There is a temporary list of IP address which have already been + called in relation to updating the IP table. +- Based on the current implementation there will 3 hops done to update + the IP table. diff --git a/Docs/ClientArchitecture.md.org b/Docs/ClientArchitecture.md.org new file mode 100644 index 0000000..daf1ca6 --- /dev/null +++ b/Docs/ClientArchitecture.md.org @@ -0,0 +1,15 @@ +* Client Module Architecture +:PROPERTIES: +:CUSTOM_ID: client-module-architecture +:END: +The Client Module interacts with the P2P module and Server Module. It is +responsible for interacting with the server module and appropriately +updating the IP table on the client side. It connects to the server +using the server's REST Apis. It is also the primary decision maker on +how the IP table is updated is on the client side. This is because each +user can have requirements like how many number of hops they would want +to do to update their IP table. Hops is the number of times the client +is going to download the IP table from different servers ,once it gets +the IP tables from the previous servers. + +[[file:images/NumOfHops.png]] [[file:images/clientmoduleArch.png]] diff --git a/Docs/ClientImplementation.md.org b/Docs/ClientImplementation.md.org new file mode 100644 index 0000000..2296f73 --- /dev/null +++ b/Docs/ClientImplementation.md.org @@ -0,0 +1,135 @@ +* Client Module Implementation +:PROPERTIES: +:CUSTOM_ID: client-module-implementation +:END: +The Client Module interacts with the P2P module and Server Module. It is +responsible for interacting with the server module and appropriately +updating the IP table on the client side. It connects to the server +using the server's REST Apis. It is also the primary decision maker on +how the IP table is updated is on the client side. This is because each +user can have requirements like how many number of hops they would want +to do to update their IP table. Hops is the number of times the client +is going to download the IP table from different servers ,once it gets +the IP tables from the previous servers. + +[[file:images/NumOfHops.png]] [[file:images/clientmoduleArch.png]] + +** Topics +:PROPERTIES: +:CUSTOM_ID: topics +:END: +1. [[#updating-the-IP-table][Updating the IP table]] +2. [[#reading-server-specifications][Reading server specifications]] +3. [[#Client-creating-and-removing-container][Client creating and + removing container]] +4. [[#Tracking-Containers][Tracking Containers]] +5. [[#Grouping-Containers][Grouping Containers]] + +This section focuses in depth on how the client module works. The client +module is incharge of communicating with different servers based on the +IP addresses provided to the user. The IP addresses are derived from +peer to peer modules. The objective here is how the client module +interacts with peer to peer module and server module. + +*** Updating the IP table +:PROPERTIES: +:CUSTOM_ID: updating-the-ip-table +:END: +The client module calls the peer to peer module to get the local IP +table initially, Based on the servers IP addresses available it calls +the speedtest function from the peer to peer module to update IP +addresses with information such as latencies, download and upload +speeds. Once this is done the client module does a Rest Api call to the +server to download its IP Table. Once the hops are done it writes the +appropriate results to the Local IP table. Once this is done it prints +out the results. To derive parameters such as current the public IP +address the url "http://ip-api.com/json/" was called. This url returns +json response of the current public IP address. This feature will be +used in the future to ensure that the user's current IP address will not +be used for a speed test. Clients IP table is updated to the server +using a form of type multipart. + +*** Reading server specifications +:PROPERTIES: +:CUSTOM_ID: reading-server-specifications +:END: +The client module calls the route /server_specs and reads the json +response. If the json response was successful then it just calls the +pretty print function which just prints the json output in the terminal. + +*** Client creating and removing container +:PROPERTIES: +:CUSTOM_ID: client-creating-and-removing-container +:END: +The client module uses the servers Rest apis to create and delete +containers. To create a container the client requires 3 parameters being +the server ip address, the number of the ports the user wants to open +and if the user wants it connected to the GPU or not. The 3 parameters +are sent as a GET request to the server and the server responds with a +json file which has information such as the container ID, ports open , +SSH username, SSH password, VNC username and VNC password. At the moment +the username and password are hard coded from the server side for both +SSH and VNC. To remove a container the client module only requires the +server IP address and the container ID. The client prints the response +from the server Rest api. + +*** Tracking Containers +:PROPERTIES: +:CUSTOM_ID: tracking-containers +:END: +Clients create docker images in multiple machines. This means if the +client (i.e user) has many containers created there needs to be a way to +track them. To track containers there is a file called +=trackcontainers.json= which tracks all the containers running. The +snippet below show a sample structure of file =trackcontainer.json=. + +#+begin_example +{ + "TrackContainer": [ + { + "ID": "", + "Container": {}, + "IpAddress": "" + } + ] +} +#+end_example + +The default path to the container tracker is +=client/trackcontainers/trackcontainers.json=. + +*** Grouping Containers +:PROPERTIES: +:CUSTOM_ID: grouping-containers +:END: +When starting a set container possibility to be able to group them. The +benefit this would be that when executing plugins the group ID would be +enough to execute plugin in a set of containers. This provides the +possibility to execute repetitive tasks in containers in a single cli +command. To store groups there is a file called +=grouptrackcontainer.json= which tracks all the groups currently present +set by the client. The snippet below show a sample structure of file +=grouptrackcontainer.json=. + +#+begin_example +{ + "Groups": [ + { + "ID": "grp", + "TrackContainer": [{client.TrackContainers struct}] + } + ] +} +#+end_example + +The default path to the container tracker is +=client/trackcontainers/grouptrackcontainer.json=. + +#+begin_quote +[!NOTE] The group id will be auto-generated and will have its own prefix +in the start which will mostly be =grp=.\\ +When a container is removed using the command. +=p2prc --rm --id =. It will be automatically +deleted from the groups it exists in. + +#+end_quote diff --git a/Docs/ConfigImplementation.md.org b/Docs/ConfigImplementation.md.org new file mode 100644 index 0000000..960093d --- /dev/null +++ b/Docs/ConfigImplementation.md.org @@ -0,0 +1,35 @@ +* Config Implementation +:PROPERTIES: +:CUSTOM_ID: config-implementation +:END: +The configuration module is responsible to store basic information of +absolute paths of files being called in the Go code. In a full-fledged +Cli the configuration file can be found in the directory /etc/ and from +there points to location such as where the IP table file is located. In +the future implementation the config file will have information such as +number of hops and other parameters to tweak and to improve the +effectiveness of the peer to peer network. The configuration module was +implemented using the library Viper. The Viper library automates +features such as searching in default paths to find out if the +configuration file is present. If the configuration file is not present +in the default paths then it auto generates the configuration file. The +configurations file can be in any format. In this project the +configuration file was generated using JSON format. + +#+begin_src json +{ + "MachineName": "pc-74-120.customer.ask4.lan", + "IPTable": "/Users/akilan/Documents/p2p-rendering-computation/p2p/iptable/ip_table.json", + "DockerContainers": "/Users/akilan/Documents/p2p-rendering-computation/server/docker/containers/", + "DefaultDockerFile": "/Users/akilan/Documents/p2p-rendering-computation/server/docker/containers/docker-ubuntu-sshd/", + "SpeedTestFile": "/Users/akilan/Documents/p2p-rendering-computation/p2p/50.bin", + "IPV6Address": "", + "PluginPath": "/Users/akilan/Documents/p2p-rendering-computation/plugin/deploy", + "TrackContainersPath": "/Users/akilan/Documents/p2p-rendering-computation/client/trackcontainers/trackcontainers.json", + "ServerPort": "8088", + "GroupTrackContainersPath": "/Users/akilan/Documents/p2p-rendering-computation/client/trackcontainers/grouptrackcontainers.json", + "FRPServerPort": "True", + "BehindNAT": "True", + "CustomConfig": null +} +#+end_src diff --git a/Docs/DesignArchtectureIntro.md.org b/Docs/DesignArchtectureIntro.md.org new file mode 100644 index 0000000..013d61b --- /dev/null +++ b/Docs/DesignArchtectureIntro.md.org @@ -0,0 +1,26 @@ +* Design Architecture +:PROPERTIES: +:CUSTOM_ID: design-architecture +:END: +This chapter focuses on architecture of the dissertation. The objective +would be to have a good understanding on the purpose of each module and +how they interact with each other. The design architecture was inspired +and based on the linux kernel design. The project is segmented into +various modules. Each module is responsible for certain tasks in the +project. The modules are highly dependent on each other hence the entire +codebase can be considered as a huge monolithic chuck which acts as its +own library. The following sub topics below talk about the main modules +and how they function with appropriate diagrams. + +*** 1. [[file:ClientArchitecture.md][Client Module]] +:PROPERTIES: +:CUSTOM_ID: client-module +:END: +*** 2. [[file:P2PArchitecture.md][P2P Module]] +:PROPERTIES: +:CUSTOM_ID: p2p-module +:END: +*** 3. [[file:ServerArchitecture.md][Server Module]] +:PROPERTIES: +:CUSTOM_ID: server-module +:END: diff --git a/Docs/DomainNameMappingsImplementation.md.org b/Docs/DomainNameMappingsImplementation.md.org new file mode 100644 index 0000000..38aca3f --- /dev/null +++ b/Docs/DomainNameMappingsImplementation.md.org @@ -0,0 +1,5 @@ +* Domain name mappings +:PROPERTIES: +:CUSTOM_ID: domain-name-mappings +:END: +This diff --git a/Docs/Implementation.md.org b/Docs/Implementation.md.org new file mode 100644 index 0000000..f797bb8 --- /dev/null +++ b/Docs/Implementation.md.org @@ -0,0 +1,66 @@ +* Implementation +:PROPERTIES: +:CUSTOM_ID: implementation +:END: +| [[file:Introduction.md][◀ Previous]] | [[file:README.md][Back to TOC]] | +|--------------------------------------+---------------------------------| + +This chapter describes how the project was built. It talks in depth of +the implementation performed to give a better understanding of the +project. + +** Programming langauge used +:PROPERTIES: +:CUSTOM_ID: programming-langauge-used +:END: +The programming language used for this project was +[[https://go.dev/][Golang]]. The reason Go lang was chosen was because +it is a compiled language. The entire codebase is just a single binary +file. When distributing to other linux distributing the only requirement +would be the binary file to run the code. It is easy to write +independant modules and be monolithic at the sametime using Go. Using +Go.mod makes it very easy to handle external libraries and modularise +code. The go.mod name for the project is +[[https://git.sr.ht/~akilan1999/p2p-rendering-computation][git.sr.ht/~akilan1999/p2p-rendering-computation]]. + +- + ** [[file:CliImplementation.md][Cli Module]] + :PROPERTIES: + :CUSTOM_ID: cli-module + :END: + +- + ** [[file:ConfigImplementation.md][Config Module]] + :PROPERTIES: + :CUSTOM_ID: config-module + :END: + +- + ** [[file:ServerImplementation.md][Server Module]] + :PROPERTIES: + :CUSTOM_ID: server-module + :END: + +- + ** [[file:ClientImplementation.md][Client Module]] + :PROPERTIES: + :CUSTOM_ID: client-module + :END: + +- + ** [[file:P2PImplementation.md][P2P Module]] + :PROPERTIES: + :CUSTOM_ID: p2p-module + :END: + +- + ** [[file:PluginImplementation.md][Plugin Module]] + :PROPERTIES: + :CUSTOM_ID: plugin-module + :END: + +- + ** [[file:GenerateImplementation.md][Generate Module]] + :PROPERTIES: + :CUSTOM_ID: generate-module + :END: diff --git a/Docs/Installation.md.org b/Docs/Installation.md.org new file mode 100644 index 0000000..5258f75 --- /dev/null +++ b/Docs/Installation.md.org @@ -0,0 +1,361 @@ +* Installation +:PROPERTIES: +:CUSTOM_ID: installation +:END: +| [[file:Introduction.md][◀ Previous]] | [[file:Abstractions.md][Next ▶]] | +|--------------------------------------+----------------------------------| + +Over here we will cover the basic steps to get the server and client +side running. + +** Latest release install +:PROPERTIES: +:CUSTOM_ID: latest-release-install +:END: +https://github.com/Akilan1999/p2p-rendering-computation/releases + +** Install from Github master branch +:PROPERTIES: +:CUSTOM_ID: install-from-github-master-branch +:END: +*** Install Go lang +:PROPERTIES: +:CUSTOM_ID: install-go-lang +:END: +The entire the implementation of this project is done using Go lang. +Thus, we need go lang to compile to code to a binary file. +[[https://golang.org/doc/install][Instructions to install Go lang]] + +*** Install Docker +:PROPERTIES: +:CUSTOM_ID: install-docker +:END: +In this project the choice of virtualization is Docker due to it's wide +usage in the developer community. In the server module we use the Docker +Go API to create and interact with the containers. + +[[https://docs.docker.com/get-docker/][Instructions to install docker]] + +[[https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker][Instructions +to install docker GPU]] + +#+begin_example +// Do ensure that the docker command does not need sudo to run +sudo chmod 666 /var/run/docker.sock +#+end_example + +*** Build Project and install project +:PROPERTIES: +:CUSTOM_ID: build-project-and-install-project +:END: +To set up the internal dependencies and build the entire go code into a +single binary + +#+begin_example +make install +#+end_example + +**** For Windows +:PROPERTIES: +:CUSTOM_ID: for-windows +:END: +To set up P2PRC on Windows, simply run this batch file. *Make sure you +are not in admin mode when running this.* + +#+begin_example +.\install.bat +#+end_example + +*** Add appropriate paths to =.bashrc= +:PROPERTIES: +:CUSTOM_ID: add-appropriate-paths-to-.bashrc +:END: +#+begin_example +export P2PRC=//p2p-rendering-computation +export PATH=//p2p-rendering-computation:${PATH} +#+end_example + +*** Set up configuration file +:PROPERTIES: +:CUSTOM_ID: set-up-configuration-file +:END: +#+begin_example +make configfile +#+end_example + +Open the config file =config.json= and add the IPv6 address if you have +one. + +*** Test if binary works +:PROPERTIES: +:CUSTOM_ID: test-if-binary-works +:END: +#+begin_example +p2prc --help +#+end_example + +**** Output: +:PROPERTIES: +:CUSTOM_ID: output +:END: +#+begin_example +NAME: + p2p-rendering-computation - p2p cli application to create and access VMs in other servers + +USAGE: + p2prc [global options] command [command options] [arguments...] + +VERSION: + + +COMMANDS: + help, h Shows a list of commands or help for one command + +GLOBAL OPTIONS: + --Server, -s Starts server (default: false) [$SERVER] + --UpdateServerList, --us Update List of Server available based on servers iptables (default: false) [$UPDATE_SERVER_LIST] + --ListServers, --ls List servers which can render tasks (default: false) [$LIST_SERVERS] + --AddServer value, --as value Adds server IP address to iptables [$ADD_SERVER] + --ViewImages value, --vi value View images available on the server IP address [$VIEW_IMAGES] + --CreateVM value, --touch value Creates Docker container on the selected server [$CREATE_VM] + --ContainerName value, --cn value Specifying the container run on the server side [$CONTAINER_NAME] + --RemoveVM value, --rm value Stop and Remove Docker container (IP:port) accompanied by container ID via --ID or --id [$REMOVE_VM] + --ID value, --id value Docker Container ID [$ID] + --Ports value, -p value Number of ports to open for the Docker Container [$NUM_PORTS] + --GPU, --gpu Create Docker Containers to access GPU (default: false) [$USE_GPU] + --Specification value, --specs value Specs of the server node [$SPECS] + --SetDefaultConfig, --dc Sets a default configuration file (default: false) [$SET_DEFAULT_CONFIG] + --NetworkInterfaces, --ni Shows the network interface in your computer (default: false) [$NETWORK_INTERFACE] + --ViewPlugins, --vp Shows plugins available to be executed (default: false) [$VIEW_PLUGIN] + --TrackedContainers, --tc View (currently running) containers which have been created from the client side (default: false) [$TRACKED_CONTAINERS] + --ExecutePlugin value, --plugin value Plugin which needs to be executed [$EXECUTE_PLUGIN] + --CreateGroup, --cgroup Creates a new group (default: false) [$CREATE_GROUP] + --Group value, --group value group flag with argument group ID [$GROUP] + --Groups, --groups View all groups (default: false) [$GROUPS] + --RemoveContainerGroup, --rmcgroup Remove specific container in the group (default: false) [$REMOVE_CONTAINER_GROUP] + --RemoveGroup value, --rmgroup value Removes the entire group [$REMOVE_GROUP] + --Generate value, --gen value Generates a new copy of P2PRC which can be modified based on your needs [$GENERATE] + --ModuleName value, --mod value New go project module name [$MODULENAME] + --PullPlugin value, --pp value Pulls plugin from git repos [$PULLPLUGIN] + --RemovePlugin value, --rp value Removes plugin [$REMOVEPLUGIN] + --help, -h show help (default: false) + --version, -v print the version (default: false) +#+end_example + +-------------- + +* Using basic commands +:PROPERTIES: +:CUSTOM_ID: using-basic-commands +:END: +*** Start as a server +:PROPERTIES: +:CUSTOM_ID: start-as-a-server +:END: +Do ensure you have Docker installed for this + +#+begin_example +p2prc -s +#+end_example + +*** View server Specification +:PROPERTIES: +:CUSTOM_ID: view-server-specification +:END: +#+begin_example +p2prc --specs= +#+end_example + +*** Run container +:PROPERTIES: +:CUSTOM_ID: run-container +:END: +use the =--gpu= if you know the other machine has a gpu. + +#+begin_example +p2prc --touch= -p --gpu +#+end_example + +*** Remove container +:PROPERTIES: +:CUSTOM_ID: remove-container +:END: +The docker id is present in the output where you create a container + +#+begin_example +p2prc --rm= --id= +#+end_example + +*** Adding servers to ip table +:PROPERTIES: +:CUSTOM_ID: adding-servers-to-ip-table +:END: +#+begin_example +p2prc --as= +#+end_example + +*** Update ip table +:PROPERTIES: +:CUSTOM_ID: update-ip-table +:END: +#+begin_example +p2prc --us +#+end_example + +*** List Servers +:PROPERTIES: +:CUSTOM_ID: list-servers +:END: +#+begin_example +p2prc --ls +#+end_example + +*** View Network interfaces +:PROPERTIES: +:CUSTOM_ID: view-network-interfaces +:END: +#+begin_example +p2prc --ni +#+end_example + +*** Viewing Containers created Client side +:PROPERTIES: +:CUSTOM_ID: viewing-containers-created-client-side +:END: +#+begin_example +p2prc --tc +#+end_example + +[[file:ClientImplementation.md#tracking-containers][read more on +tracking containers]] + +*** Running plugin +:PROPERTIES: +:CUSTOM_ID: running-plugin +:END: +#+begin_example +p2prc --plugin --id +#+end_example + +*** Create group +:PROPERTIES: +:CUSTOM_ID: create-group +:END: +#+begin_example +p2prc --cgroup +#+end_example + +*** Add container to group +:PROPERTIES: +:CUSTOM_ID: add-container-to-group +:END: +#+begin_example +p2prc --group --id +#+end_example + +*** View groups +:PROPERTIES: +:CUSTOM_ID: view-groups +:END: +#+begin_example +p2prc --groups +#+end_example + +*** View specific group +:PROPERTIES: +:CUSTOM_ID: view-specific-group +:END: +#+begin_example +p2prc --group +#+end_example + +*** Delete container from group +:PROPERTIES: +:CUSTOM_ID: delete-container-from-group +:END: +#+begin_example +p2prc --rmcgroup --group --id +#+end_example + +*** Delete entire group +:PROPERTIES: +:CUSTOM_ID: delete-entire-group +:END: +#+begin_example +p2prc --rmgroup +#+end_example + +[[file:ClientImplementation.md#Grouping-Containers][read more on +grouping containers]] ### Extending usecase of P2PRC (Requires a go +compiler to run) + +#+begin_example +p2prc --gen --mod +#+end_example + +[[file:GenerateImplementation.md][read more about the generate module]] + +*** Pulling plugin from a remote repo +:PROPERTIES: +:CUSTOM_ID: pulling-plugin-from-a-remote-repo +:END: +#+begin_example +p2prc --pp +#+end_example + +*** Deleting plugin from the plugin directory +:PROPERTIES: +:CUSTOM_ID: deleting-plugin-from-the-plugin-directory +:END: +#+begin_example +p2prc --rp +#+end_example + +*** Added custom metadata about the current node +:PROPERTIES: +:CUSTOM_ID: added-custom-metadata-about-the-current-node +:END: +#+begin_example +p2prc --amd "custom metadata" +#+end_example + +-------------- + +* Using Plugins +:PROPERTIES: +:CUSTOM_ID: using-plugins +:END: +This feature is still Under Development: +[[file:PluginImplementation.md][Read more on the implementation]] + +**** Dependencies +:PROPERTIES: +:CUSTOM_ID: dependencies +:END: +- Ansible: + - Debian/ubuntu: =sudo apt install ansible= + - Others: + [[https://ansible-tips-and-tricks.readthedocs.io/en/latest/ansible/install/][Installation + link]] + +**** Run Test Cases +:PROPERTIES: +:CUSTOM_ID: run-test-cases +:END: +- Generate Test Case Ansible file + - =make testcases= +- Enter inside plugin directory and run tests. + +#+begin_quote +[!NOTE] That docker needs to installed and needs to run without sudo. +Refer the section [[#install-docker][Install Docker]]. - =cd plugin= - +=go test .= + +#+end_quote + +-------------- + +*** Next Chapter: [[file:Abstractions.md][Abstractions]] +:PROPERTIES: +:CUSTOM_ID: next-chapter-abstractions +:END: diff --git a/Docs/Introduction.md.org b/Docs/Introduction.md.org new file mode 100644 index 0000000..5ded787 --- /dev/null +++ b/Docs/Introduction.md.org @@ -0,0 +1,48 @@ +* Chapter 1: Introduction +:PROPERTIES: +:CUSTOM_ID: chapter-1-introduction +:END: +| [[file:README.md][◀ Back to TOC]] | [[file:Installation.md][Next ▶]] | +|-----------------------------------+----------------------------------| + +** Abstract +:PROPERTIES: +:CUSTOM_ID: abstract +:END: +This project focuses on creating a framework on running heavy tasks that +a regular computer cannot run easily such as graphically demanding video +games, rendering 3D animations , protein folding simulations. In this +project the major focus will not be on the financial incentive part. A +peer to peer network will be created to help run tasks decentrally, +increasing bandwidth for running tasks. To ensure the tasks in the peer +to peer network do not corrupt the server 0S (Operating System), they +will be executed in a virtual environment in the server. + +The main aim of this project was to create a custom peer to peer +network. The user acting as the client has total flexibility on how to +batch the tasks and the user acting as the server has complete +flexibility on tracking the container's usages and killing the +containers at any point of time. + +** Motivation +:PROPERTIES: +:CUSTOM_ID: motivation +:END: +Many of the users rely on our PC / Laptop or servers that belong to a +server farm to run heavy tasks and with the demand of high creativity +requires higher computing power. Buying a powerful computer every few +years to run a bunch of heavy tasks which are not executed as frequently +to reap the benefits can be inefficient utilization of hardware. On the +other end, renting servers to run these heavy tasks can be really +useful. Ethically speaking this is leading to monopolisation of +computing power similar to what is happening in the web server area. By +using peer to peer principles it is possible to remove the +monopolisation factor and increase the bandwidth between the client and +server. + +-------------- + +*** Next Chapter: [[file:Installation.md][Installation]] +:PROPERTIES: +:CUSTOM_ID: next-chapter-installation +:END: diff --git a/Docs/NAT-Traveral.md.org b/Docs/NAT-Traveral.md.org new file mode 100644 index 0000000..d96ea8b --- /dev/null +++ b/Docs/NAT-Traveral.md.org @@ -0,0 +1,50 @@ +* NAT Traversal +:PROPERTIES: +:CUSTOM_ID: nat-traversal +:END: +P2PRC currently supports TURN for NAT traversal. + +** TURN +:PROPERTIES: +:CUSTOM_ID: turn +:END: +The current TURN implementation used is FRP. The TURN server is also +required when a P2PRC node is acting as a Server. The TURN server is +determined based on the Node with the least amount of latency based on +the Nodes available on the IPTable. Once a TURN server is determined +there are 2 actions performed. The first one is =/FRPPort= to the TURN +server to receive a port which is used to generate the external port +from the TURN server. The flow below describes the workflow. + +*** Client mode +:PROPERTIES: +:CUSTOM_ID: client-mode +:END: +- Call =/FRPPort= + +#+begin_example +http://:/FRPport +#+end_example + +- Call the TURN server in the following manner. The following is a + sample code snippet below. + +#+begin_src go +import ( + "github.com/Akilan1999/p2p-rendering-computation/p2p/frp" +) + +func main() { + serverPort, err := frp.GetFRPServerPort("http://" + + ":" + lowestLatencyIpAddress.ServerPort) + if err != nil { + return nil, err + } + // Create 1 second delay to allow FRP server to start + time.Sleep(1 * time.Second) + // Starts FRP as a client with + proxyPort, err := frp.StartFRPClientForServer(, serverPort, ) + if err != nil { + return nil, err + } +} +#+end_src diff --git a/Docs/P2P-testing.md.org b/Docs/P2P-testing.md.org new file mode 100644 index 0000000..67c5be9 --- /dev/null +++ b/Docs/P2P-testing.md.org @@ -0,0 +1,136 @@ +* Testing P2P network +:PROPERTIES: +:CUSTOM_ID: testing-p2p-network +:END: +The objective would be to test the p2p network, and the effectiveness of +updating the ip tables. The objective of would be to give the impression +to the client and server of a Zero configuration setting. For testing +there will be a test network set. In the testing scenario all will be +client and server because the IP table does not store clients IP +addresses. At current number of hopes would be 3 as default. + +*** Test Network Scenario 1 +:PROPERTIES: +:CUSTOM_ID: test-network-scenario-1 +:END: +The test network consists of 5 nodes acting as a client and server. The +objective would be to have the entire IP table Updated in each node with +interacting with only 1 node once. Each node has knowledge of one node +only. + +[[https://user-images.githubusercontent.com/31743758/115069627-e4aa8c80-9f04-11eb-8402-706a3407f0e8.png]] +Fig 1.0 Visual Representation of testnet scenario 1 + +**** Result +:PROPERTIES: +:CUSTOM_ID: result +:END: +All nodes except node 1 where able to have information of IP addresses +in the test net. This was due to the reason of 3 hops set as default. +Node 1 had in it's IP table IP addresses of Node 2, Node 3, Node 4. Once +the number of hops was set to 4 objective of the test was acheived. + +*** Test Network Scenario 2 +:PROPERTIES: +:CUSTOM_ID: test-network-scenario-2 +:END: +The second test network has a scenario of a single peer which all the +other nodes connect too. The scenario being when the other nodes connect +to the single server they download information about nodes that have +connected to the server node before. + +*** Testing Broadcast Module +:PROPERTIES: +:CUSTOM_ID: testing-broadcast-module +:END: +For testing the broadcast module 2 types of servers will be tested. One +with a CPU only , another one with a CPU and GPU. The expected result +being that the appropriate results are visible. + +**** Results (CPU and GPU): +:PROPERTIES: +:CUSTOM_ID: results-cpu-and-gpu +:END: +#+begin_example +{ + "Hostname": "akilan-Lenovo-IdeaPad-Y510P", + "Platform": "ubuntu", + "CPU": "Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz", + "RAM": 7872, + "Disk": 937367, + "GPU": { + "DriveVersion": "390.141", + "Gpu": { + "GpuName": "GeForce GT 755M", + "BiosVersion": "80.07.A8.00.0F", + "FanSpeed": "N/A", + "Utilization": { + "GpuUsage": "N/A", + "MemoryUsage": "N/A" + }, + "Temperature": { + "GpuTemp": "66 C" + }, + "Clock": { + "GpuClock": "N/A", + "GpuMemClock": "N/A" + } + } + } +} +#+end_example + +At the moment of the current implementation v1.0. Nvidia GPU are only +compatible. As the Go code calls the command =nvidia-smi= to get +information about the GPU available. + +**** Results (CPU only) +:PROPERTIES: +:CUSTOM_ID: results-cpu-only +:END: +#+begin_example +{ + "Hostname": "sv-t1.small.x86-01", + "Platform": "ubuntu", + "CPU": "Intel(R) Atom(TM) CPU C2750 @ 2.40GHz", + "RAM": 7944, + "Disk": 138793, + "GPU": null +} +#+end_example + +As the =nvidia-smi= interface was not detected it only broadcasts the +CPU specs available. + +*** SpeedTests +:PROPERTIES: +:CUSTOM_ID: speedtests +:END: +The speed test has 3 parameters which are Ping , upload and download. +The tests check if the results returned are approximately correct. The +ping at the moment returns the correct result. The upload and download +returned are inccorect at the moment, This is due incorrect +implementation in for timer and will be patched in future versions. + +*** Unit tests +:PROPERTIES: +:CUSTOM_ID: unit-tests +:END: +All functions implemented on the P2P module returns type error. The +units test call certain functions and check if the functions return an +error or not. This proved sufficient as the point of the units tests was +code coverage to check if certain functions return an error. + +**** Functions tested +:PROPERTIES: +:CUSTOM_ID: functions-tested +:END: +This sections talks about the function called and represents code +coverage. + +1. =TestServer_SpeedTest=: Function called LocalSpeedTestIpTable() +2. =TestReadIpTable=: Function called ReadIpTable() + +The P2P module has a 100% code coverage in unit tests as both the unit +tests call directly or call within the function all the functions used +in the P2P module. diff --git a/Docs/P2P.md.org b/Docs/P2P.md.org new file mode 100644 index 0000000..8d34d62 --- /dev/null +++ b/Docs/P2P.md.org @@ -0,0 +1,45 @@ +* P2P (Peer to Peer module) +:PROPERTIES: +:CUSTOM_ID: p2p-peer-to-peer-module +:END: +In this repository the P2P module has been designed from sratch at the +point of this implementation. +[[https://pkg.go.dev/git.sr.ht/~akilan1999/p2p-rendering-computation@v0.0.0-20210404191839-6a046babcb02/p2p][More +about function implementation]] + +** Terminology +:PROPERTIES: +:CUSTOM_ID: terminology +:END: +1. IPTable: Refers to a json file which stores information about the + current servers avaliable with the speedtest results ran from the + Node that triggered it. + +#+begin_example +{ +"ip_address": [ + { + "ipv4": "localhost", + "latency": 14981051, + "download": 8142.122540206258, + "upload": 3578.766512629995, + } +] +} +#+end_example + +** Responsibility +:PROPERTIES: +:CUSTOM_ID: responsibility +:END: +- To ensure the IP table has nodes which are pingable +- Taking to nodes behind NAT. [[file:NAT-Traversal][More about the + implementation]]... + +#+begin_quote +[!NOTE] If you are running in server mode it is recommended to use +[[https://routerguide.net/when-and-how-to-setup-dmz-host-for-home-use/][DMZ]] +to bypass the +[[https://en.wikipedia.org/wiki/Network_address_translation][NAT]]. + +#+end_quote diff --git a/Docs/P2PArchitecture.md.org b/Docs/P2PArchitecture.md.org new file mode 100644 index 0000000..c7a3ad4 --- /dev/null +++ b/Docs/P2PArchitecture.md.org @@ -0,0 +1,16 @@ +* P2P Module Architecture +:PROPERTIES: +:CUSTOM_ID: p2p-module-architecture +:END: +The P2P module (i.e Peer to Peer Module) is responsible for storing the +IP table and interacting with the IP table. In the following +implementation of the P2P module ,the IP table stores information about +servers available in the network. The other functionality the P2P module +takes care of is doing the appropriate speed tests to the servers in the +IP table. This is for informing the users about nodes which are close by +and nodes which have quicker uploads and downloads speeds. The module is +responsible to ensure that there are no duplicate server IPs in the IP +table and to remove all server IPs which are not pingable. + +#+caption: UML diagram of P2P module +[[file:images/p2pmoduleArch.png]] diff --git a/Docs/P2PImplementation.md.org b/Docs/P2PImplementation.md.org new file mode 100644 index 0000000..344fff8 --- /dev/null +++ b/Docs/P2PImplementation.md.org @@ -0,0 +1,111 @@ +* P2P Module Implementation +:PROPERTIES: +:CUSTOM_ID: p2p-module-implementation +:END: +The P2P module (i.e Peer to Peer Module) is responsible for storing the +IP table and interacting with the IP table. In the following +implementation of the P2P module ,the IP table stores information about +servers available in the network. The other functionality the P2P module +takes care of is doing the appropriate speed tests to the servers in the +IP table. This is for informing the users about nodes which are close by +and nodes which have quicker uploads and downloads speeds. The module is +responsible to ensure that there are no duplicate server IPs in the IP +table and to remove all server IPs which are not pingable. + +#+caption: UML diagram of P2P module +[[file:images/p2pmoduleArch.png]] + +The peer to peer implementation was built from scratch. This is because +other peer to peer libraries were on the implementation of the +Distributed hash table. At the current moment all those heavy features +are not needed because the objective is to search and list all possible +servers available. The limitation being that to be a part of the network +the user has to know at least 1 server. The advantage of building from +scratch makes the module super light and possibility for custom +functions and structs. The sub topics below will mention the +implementations of each functionality in depth. + +** IP Table +:PROPERTIES: +:CUSTOM_ID: ip-table +:END: +The ip table file is a json as the format with a list of servers ip +addresses, latencies, downloads and uploads speeds. The functions +implemented include read file, write file and remove duplicate IP +addresses. The remove duplicate IP address function exists because +sometimes servers IP tables can have the same ip addresses as what the +client has. The path of the IP table json file is received from the +configuration module. + +#+begin_src json +{ + "ip_address": [ + { + "ipv4": "", + "latency": "", + "download": "", + "upload": "" + "port no": "", + } + ] +} +#+end_src + +*** Latency +:PROPERTIES: +:CUSTOM_ID: latency +:END: +The latency is measured in milliseconds. The route /server_info is +called from the server and time it takes to provide a json response is +recorded. + +** NAT Traversal +:PROPERTIES: +:CUSTOM_ID: nat-traversal +:END: +P2PRC currently supports TURN for NAT traversal. + +** TURN +:PROPERTIES: +:CUSTOM_ID: turn +:END: +The current TURN implementation used is FRP. The TURN server is also +required when a P2PRC node is acting as a Server. The TURN server is +determined based on the Node with the least amount of latency based on +the Nodes available on the IPTable. Once a TURN server is determined +there are 2 actions performed. The first one is =/FRPPort= to the TURN +server to receive a port which is used to generate the external port +from the TURN server. The flow below describes the workflow. + +*** Client mode +:PROPERTIES: +:CUSTOM_ID: client-mode +:END: +- Call =/FRPPort= + +#+begin_example +http://:/FRPport +#+end_example + +- Call the TURN server in the following manner. The following is a + sample code snippet below. + +#+begin_src go +import ( + "github.com/Akilan1999/p2p-rendering-computation/p2p/frp" +) + +func main() { + serverPort, err := frp.GetFRPServerPort("http://" + + ":" + lowestLatencyIpAddress.ServerPort) + if err != nil { + return nil, err + } + // Create 1 second delay to allow FRP server to start + time.Sleep(1 * time.Second) + // Starts FRP as a client with + proxyPort, err := frp.StartFRPClientForServer(, serverPort, ) + if err != nil { + return nil, err + } +} +#+end_src diff --git a/Docs/PluginImplementation.md.org b/Docs/PluginImplementation.md.org new file mode 100644 index 0000000..cca9688 --- /dev/null +++ b/Docs/PluginImplementation.md.org @@ -0,0 +1,213 @@ +* Plugin Module Implementation +:PROPERTIES: +:CUSTOM_ID: plugin-module-implementation +:END: +** Topics +:PROPERTIES: +:CUSTOM_ID: topics +:END: +1. [[#introduction][Introduciton]] +2. [[#site-File-Template][Site.yml]] +3. [[#hosts-file][Host]] +4. [[#description-file][Description]] +5. [[#automatic-port-allocations][Automatic port allocations]] +6. [[#sample-plugins-implemented][Sample plugins implemented]] + +** Introduction +:PROPERTIES: +:CUSTOM_ID: introduction +:END: +The plugin module is designed to ensure clients can execute instructions +in a declarative manner across different containers created. This means +the user (i.e client) needs to write the instruction only once, and +these instructions can be executed across different nodes in a +repetitive manner. + +In the scenario of this project Ansibles will be used as the way the +users can create these instructions. + +- [[file:Installation.md#Using-Plugins][Setup instruction]] + +The plugin module introduces a new path to the config file known as +pluginpath. This path by defaults points to =${P2PRC}/plugin/deploy=. +Any file/folder inside =plugin/deploy= is part of the .gitginore. +Plugins are detected by folder names inside the =plugin/deploy=. + +#+begin_example +plugin +|___ Deploy + |___ + |___ site.yml + |___ hosts + |___ ports.json + |___ description.txt + . + . + . + n: n number of plugins possible +#+end_example + +** Site File Template +:PROPERTIES: +:CUSTOM_ID: site-file-template +:END: +The site file is also known as the Ansible playbook and is incharge of +executing instructions in a declarative manner. The below example +specifies how to make one. + +#+begin_example +- hosts: all + + tasks: + - name: + + debug: + msg: +#+end_example + +Read more about ansible tasks: +https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html#about-playbooks + +** Hosts file +:PROPERTIES: +:CUSTOM_ID: hosts-file +:END: +hosts file is also known as the inventory file. This file consists of +all the information required to connect to other nodes to execute +Ansible instructions. In this project this file needs to be set in a +certain configuration because the go code or binary will populate this +file automatically with the appropriate information required to connect +to local or remote containers. + +#+begin_quote +[!NOTE] Add as exactly specified below: + +#+begin_example +all: + vars: + ansible_python_interpreter: /usr/bin/python3 // Path to your python 3 interpreter +main: + hosts: + host1: + // Note: These values will be automatically overwritten + // by the Go functions + ansible_host: 0.0.0.0 + ansible_port: 39269 + ansible_user: master + ansible_ssh_pass: password + ansible_sudo_pass: password +#+end_example + +#+end_quote + +** Ports.json +:PROPERTIES: +:CUSTOM_ID: ports.json +:END: +The =ports.json= file is intended to mention the number of ports +required by the plugin. + +#+begin_example +{ + "NumOfPorts": +} +#+end_example + +** Description file +:PROPERTIES: +:CUSTOM_ID: description-file +:END: +This is a simple text file used to describe what the module does. When +the client is looking at various commands via the ClI. The description +is displayed along-side the plugin name. + +Ex: When the flag =--ViewPlugins= or =--vp= is called + +#+begin_example +{ + "PluginsDetected": [ + { + "FolderName": "", + "PluginDescription": "" + } + ] +} +#+end_example + +** Automatic port allocations +:PROPERTIES: +:CUSTOM_ID: automatic-port-allocations +:END: +P2PRC would be in-charge to set to the ports to various TCP ports +opened. Due to this implementation the plugin being executed is copied +to the tmp directory with a unique UUID. + +#+begin_example +Command: ls /tmp +output: Semantic _ +2e6d76c4-0ed1-4b55-9385-79a58d4f0492_p2prc-vscode-browser +7b631e08-62ee-4c1c-a2a4-c05857b9aa7d_p2prc-vscode-browser +#+end_example + +Once the copy of the plugin is added to the /tmp directory the site.yml +file inside the appropriate yaml is modified with the appropriate ports +assigned to the container. + +*** Ex: +:PROPERTIES: +:CUSTOM_ID: ex +:END: +1. Create container called c1 with an automatic generated TCP port 3313 + (external) - 3313 (internal) +2. Assumption of plugin p1 exists. p1 has one server which needs to be + mapped to a free open TCP port in container c1. Below shows an + implementation of a sample site.yml file. + +#+begin_example +--- +- hosts: all + tasks: + - name: start vscode code server + shell: sh server.sh 0.0.0.0:{{index . 0}} +#+end_example + +Notice there is the following {{index . 0}}. {{index . 0}} does not +belong to Ansible but rather is a way to mention where to add the +external free port of the container. We use the golang +[[https://pkg.go.dev/text/template][template library]] to parse and +populate the site.yml with the appropriate open ports. An array of ints +which consists of open free ports are sent to the site.yml. 0 in +{{index . 0}} refers to the index in the int array passed on. + +After the port is automatically it's ready to run ! + +#+begin_example +--- +- hosts: all + tasks: + - name: start vscode code server + shell: sh server.sh 0.0.0.0:3313 +#+end_example + +*** Sample plugins implemented: +:PROPERTIES: +:CUSTOM_ID: sample-plugins-implemented +:END: +- [[https://github.com/Akilan1999/p2prc-vscode-browser][VSCode Plugin]] + +** Pull Plugins +:PROPERTIES: +:CUSTOM_ID: pull-plugins +:END: +The following allows us to pull plugins from a remote git repository and +store them in the default plugins directory. The implementation uses a +Go git library to pull the git repo and automatically save it as a +folder in the plugin path. + +** Delete Plugins +:PROPERTIES: +:CUSTOM_ID: delete-plugins +:END: +We delete the plugin folder based on the plugin name provided as an +argument on the cli command. Once the folder is deleted, the plugin +manager automatically knows that the plugin does not exist anymore. diff --git a/Docs/README.md.org b/Docs/README.md.org new file mode 100644 index 0000000..1659f36 --- /dev/null +++ b/Docs/README.md.org @@ -0,0 +1,18 @@ +* Table of contents +:PROPERTIES: +:CUSTOM_ID: table-of-contents +:END: +1. [[file:Introduction.md][Introduction]] +2. [[file:Installation.md][Installation]] +3. [[file:Abstractions.md][Abstractions]] +4. [[file:Implementation.md][Implementation]] + 1. [[file:ClientImplementation.md][Client Module]] + 2. [[file:P2PImplementation.md][P2P Module]] + 3. [[file:ServerImplementation.md][Server Module]] + 4. [[file:ConfigImplementation.md][Config Module]] + 5. [[file:CliImplementation.md][Cli Module]] + 6. [[file:PluginImplementation.md][Plugin Module]] + 7. [[file:Bindings.md][Language bindings]] + 8. [[file:Bindings.md][Domain name mappings]] +5. Language bindings + 1. [[file:haskell/][Haskell]] diff --git a/Docs/ServerArchitecture.md.org b/Docs/ServerArchitecture.md.org new file mode 100644 index 0000000..caf1cca --- /dev/null +++ b/Docs/ServerArchitecture.md.org @@ -0,0 +1,14 @@ +* Server Module Architecture +:PROPERTIES: +:CUSTOM_ID: server-module-architecture +:END: +The server module takes care of setting and removing the virtualization +environment (i.e containers) for accessing and doing the appropriate +computation. It also interacts with the peer to peer module to update +the IP table on the server side. The server module accesses information +regarding CPU and GPU specifications of the machine running the server +module. To do Speed tests the server has routes which allows it to +upload and download a 50mb. + +#+caption: UML diagram of server module +[[file:images/servermoduleArch.png]] diff --git a/Docs/ServerImplementation.md.org b/Docs/ServerImplementation.md.org new file mode 100644 index 0000000..5e7713c --- /dev/null +++ b/Docs/ServerImplementation.md.org @@ -0,0 +1,278 @@ +* Server Module Implementation +:PROPERTIES: +:CUSTOM_ID: server-module-implementation +:END: +This section focuses on an in-depth understanding of the server module +implementation. To understand the architecture of the server module +refer. The server module can be split into various sections. Each +section will provide information on how a certain feature works. + +The server module takes care of setting and removing the virtualization +environment (i.e containers) for accessing and doing the appropriate +computation. It also interacts with the peer to peer module to update +the IP table on the server side. The server module accesses information +regarding CPU and GPU specifications of the machine running the server +module. To do Speed tests the server has routes which allows it to +upload and download a 50mb. + +#+caption: UML diagram of server module +[[file:images/servermoduleArch.png]] + +** Web framework +:PROPERTIES: +:CUSTOM_ID: web-framework +:END: +The web framework used for the server module is called Gin. The reason +Gin was chosen is due to its wide use and strong documentation available +on the official github repository. The default port used is 8088. For +version 1.0 of the project ,the server needs to keep port 8088 open to +ensure that other clients and servers can detect it. The possible +requests available are GET and POST for this implementation. The +possible responses are either a string or json response or a file. In +the majority of routes a string response refers to an error when calling +the following routes. The following sub topics below will talk about the +route implemented: + +*** /server_info +:PROPERTIES: +:CUSTOM_ID: server_info +:END: +This route is responsible to get information about the specifications of +the server. The response of this route is in json if the call was +successful. + +*** /50 +:PROPERTIES: +:CUSTOM_ID: section +:END: +This route is responsible for returning a randomly generated 50mb file. +This is used to calculate the download speed from the p2p module. + +*** /IpTable +:PROPERTIES: +:CUSTOM_ID: iptable +:END: +This route is a POST request that is responsible to update the server IP +table based on the IP table the client provides. Once the server gets +the IP table it checks if the client is also a server. This is done by +calling the url http://:8088/server_info. If the server_info route from +the client responds back with computer specifications of the client. +Then the server initially appends the clients IP to the struct. After +that the IP table received from the client is uploaded to the struct. +Once this is done the server passes the struct to the peer to peer +module function. The peer to peer module function will return the back +with the new struct with the valid server nodes. The server responds +back to the new struct as a json format. If a string is present in the +response then there is probably an error on the server side. + +*** /startcontainer +:PROPERTIES: +:CUSTOM_ID: startcontainer +:END: +This route takes in a GET request with the number of TCP ports to open +and checks whether the docker container should be hooked to the GPU or +not. This route talks to the docker module implemented as a sub module +in the server module. More information on the docker module in section +5.4.3. This route calls docker the module to start the container for the +client. The docker module returns back a struct. This struct is returned +back to the client as the json response. This struct consists of +information such as docker id, ports numbers open , information +regarding SSH and VNC connections to the docker container created when +the client created this request. + +*** /RemoveContainer +:PROPERTIES: +:CUSTOM_ID: removecontainer +:END: +This route takes in a GET request as the container ID. Based on the +container ID provided ,it calls the docker module which deletes the +container. If the deletion is successful it returns back a string which +says success. + +** Server information/ Specification +:PROPERTIES: +:CUSTOM_ID: server-information-specification +:END: +This section provides information on how the server specifications are +read. There are 2 major implementations. The first implementation +mentions how basic information such as RAM usage, CPU specification are +detected and the second implementation mentions how the GPU drivers are +detected and information is extracted. The client has to assume that the +server is using default docker settings in terms of CPU cycles and other +parameters. + +*** Basic Information +:PROPERTIES: +:CUSTOM_ID: basic-information +:END: +The file name for these functions is called gopsutil.go. This codebase +uses the library gopsutil. Gopsutil has various packages or modules +within the library which have functions implemented to get system +information. The following information is stored in a struct and the +function returns that struct. + +#+begin_src go +type SysInfo struct { + Hostname string `bson:hostname` + Platform string `bson:platform` + CPU string `bson:cpu` + RAM uint64 `bson:ram` + Disk uint64 `bson:disk` + GPU *Query `xml: GpuInfo` +} +#+end_src + +*** GPU Information +:PROPERTIES: +:CUSTOM_ID: gpu-information +:END: +The file name for these functions is called GPU.go. This codebase checks +if the Nvidia driver exists and returns the driver information. To do +this a shell command called nvidia-smi is executed. This shell command +is executed with a --xml as flag to ensure that the output is in the XML +format. If there is an output as a xml format, that means there is an +nvidia driver installed, and the function just reads the output and +stores it to the struct and returns the GPU information. + +#+begin_src go + +type Query struct { + DriveVersion string `xml:"driver_version"` + Gpu Gpu `xml:"gpu"` +} + +type Gpu struct{ + GpuName string `xml:"product_name"` + BiosVersion string `xml:"vbios_version"` + FanSpeed string `xml:"fan_speed"` + Utilization GpuUtilization `xml:"utilization"` + Temperature GpuTemperature `xml:"temperature"` + Clock GpuClock `xml:"clocks"` +} + +type GpuUtilization struct { + GpuUsage string `xml:"gpu_util"` + MemoryUsage string `xml:"memory_util"` +} + +type GpuTemperature struct { + GpuTemp string `xml:"gpu_temp"` +} + +type GpuClock struct { + GpuClock string `xml:"graphics_clock"` + GpuMemClock string `xml:"mem_clock"` +} +#+end_src + +** Docker Module +:PROPERTIES: +:CUSTOM_ID: docker-module +:END: +This section provides information on how the server module interacts +with the docker containers. The server calls 2 routes which either +creates or removes the docker container. Docker has a huge advantage +because it takes less than 20 seconds to spin up a new container once +it's built and executed at least once. For docker operations a separate +module/package has been created. The following subtopics will provide +more information on how this package works. + +*** Docker Api +:PROPERTIES: +:CUSTOM_ID: docker-api +:END: +For this the api has been taken from the official docker repository. To +be more specific it is the client module in the official docker +repository. Docker was built using Go. During this project Docker +functions could be directly called from the docker repository. The +Docker api initially ensures that it can detect the docker environment +variables. Once detected, it can execute various functions from the +docker client module. The reason the docker api was selected was to +detect and handle errors better. + +*** Docker Image +:PROPERTIES: +:CUSTOM_ID: docker-image +:END: +The docker image used to spin up the containers is called +ConSol/docker-headless-vnc-container. The following container was +modified to open SSH ports for an SSH connection. The following docker +image runs ubuntu 16. The reason this image was chosen as a default is +because if the client wants to access the container in the form of a +desktop environment. This image would allow the client to do so from +just a browser. + +*** Build container +:PROPERTIES: +:CUSTOM_ID: build-container +:END: +This function pulls the docker image locally and builds the image. +Initially there is a timeout function to ensure that building the image +does not take too long to build. The next phase would be based on the +path to get the DockerFile. The tag name of the container is set as +p2p-ubuntu as default. Once the following is set then the docker build +command is executed. + +*** Run container +:PROPERTIES: +:CUSTOM_ID: run-container +:END: +After building the container it needs to be executed for the user to +access the container and do certain operations. The docker +package/module has a function to do this. The function takes in the +docker environment as a parameter and also the docker struct. The docker +struct has information such as the TCP ports which are supposed to be +open and whether the docker container should have the GPU hooked to it +or not. Based on the appropriate information provided ,the docker image +gets started. The Image gets started by interacting with the docker +client modules. When hooking the GPU the docker run command is called +from the shell. This is because the docker Api does not support the GPU +module yet. When the container is executed for the first time it takes +more than 10 minutes to build. From the second time onwards it takes +only 10 seconds to run. + +*** Stop and remove container +:PROPERTIES: +:CUSTOM_ID: stop-and-remove-container +:END: +This implementation here ensures that the docker is stopped, and the +container is removed. This is to ensure it does not utilize server +resources when it is not being used, or the task that is intended to be +executed is complete. To run this function all that is needed is the +docker container ID. If the function is successful it returns a string +that says success. + +*** Ports json file +:PROPERTIES: +:CUSTOM_ID: ports-json-file +:END: +This file will help map internal ports inside a container to external +ports inside a container. A common example would be the SSH port which +is port 22 inside the docker container and is mapped to random TCP port +outside container so that any external machines can directly connect +into the container. The below representation mentions of where the +ports.json file is located and also the format of that file. + +#+begin_example +|_ + |_ Dockerfile + |_ description.txt + |_ ports.json // The ports file +#+end_example + +Format of the ports.json file + +#+begin_example +{ + "Port": [ + { + "PortName": "", + "InternalPort": , + "Type": "", + "ExternalPort": , + "IsUsed": "", + "Description": "" + }, ... n + ] +} +#+end_example