me/p2p-rendering-computation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

did basic conversion to org mode for the docs folder

Browse Source
This commit is contained in:
Akilan Selvacoumar
2025-01-18 17:30:10 +00:00
parent f38b6d2871
commit ca46b20be1
21 changed files with 1844 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
Docs/Abstractions.md.org Normal file
View File

@@ -0,0 +1,35 @@
* Abstractions
:PROPERTIES:
:CUSTOM_ID: abstractions
:END:
| [[file:Installation.md][◀ Previous]] | [[file:Implementation.md][Next ▶]] |
|--------------------------------------+------------------------------------|
The Abstractions package consists of black-boxed functions for P2PRC.
** Functions
:PROPERTIES:
:CUSTOM_ID: functions
:END:
- =Init(<Project name>)=: Initializes P2PRC with all the needed
configurations.
- =Start()=: Starts p2prc as a server and makes it possible to extend by
adding other routes and functionality to P2PRC.
- =MapPort(<port no>)=: On the local machine the port you want to export
to world.
- =StartContainer(<ip address>)=: The machine on the p2p network where
you want to spin up a docker container.
- =RemoveContainer(<ip address>,<container id>)=: Terminate container
based on the IP address and container name.
- =GetSpecs(<ip address>)=: Get specs of a machine on the network based
on the IP address.
- =ViewIPTable()=: View the IP table which about nodes in the network.
- =UpdateIPTable()=: Force update IP table to learn about new nodes
faster.
--------------
*** Next Chapter: [[file:Implementation.md][Implementation]]
:PROPERTIES:
:CUSTOM_ID: next-chapter-implementation
:END:

180
Docs/Bindings.md.org Normal file
View File

@@ -0,0 +1,180 @@
* Language Bindings
:PROPERTIES:
:CUSTOM_ID: language-bindings
:END:
[[https://en.wikipedia.org/wiki/Language_binding][Language bindings]]
refers to wrappers to bridge 2 programming languages. This is used in
P2PRC to extend calling P2PRC functions in other programming languages.
Currently this is done by generating =.so= and =.h= from the Go
compiler.
** How to build shared object files
:PROPERTIES:
:CUSTOM_ID: how-to-build-shared-object-files
:END:
**** The easier way
:PROPERTIES:
:CUSTOM_ID: the-easier-way
:END:
#+begin_src sh
# Run
make sharedObjects
#+end_src
**** Or the direct way
:PROPERTIES:
:CUSTOM_ID: or-the-direct-way
:END:
#+begin_src sh
# Run
cd Bindings && go build -buildmode=c-shared -o p2prc.so
#+end_src
**** If successfully built:
:PROPERTIES:
:CUSTOM_ID: if-successfully-built
:END:
#+begin_src sh
# Enter into the Bindings directory
cd Bindings
# List files
ls
# Find files
p2prc.h p2prc.so
#+end_src
** Workings under the hood
:PROPERTIES:
:CUSTOM_ID: workings-under-the-hood
:END:
Below are a sample set of commands to open the bindings implementation.
#+begin_example
# run
cd Bindings/
# list files
ls
# search for file
Client.go
#+end_example
*** In Client go
:PROPERTIES:
:CUSTOM_ID: in-client-go
:END:
There a few things to notice which are different from your standard Go
programs:
**** 1. We import "C" which means [[https://pkg.go.dev/cmd/cgo][Cgo]] is required.
:PROPERTIES:
:CUSTOM_ID: we-import-c-which-means-cgo-is-required.
:END:
#+begin_src go
import "C"
#+end_src
**** 2. All functions which are required to be called from other programming languages have comment such as.
:PROPERTIES:
:CUSTOM_ID: all-functions-which-are-required-to-be-called-from-other-programming-languages-have-comment-such-as.
:END:
#+begin_src go
//export <function name>
// ------------ Example ----------------
// The function below allows to externally
// to call the P2PRC function to start containers
// in a specific node in the know list of nodes
// in the p2p network.
// Note: the comment "//export StartContainer".
//export StartContainer
func StartContainer(IP string) (output *C.char) {
container, err := client.StartContainer(IP, 0, false, "", "")
if err != nil {
return C.CString(err.Error())
}
return ConvertStructToJSONString(container)
}
#+end_src
**** 3. While looking through the file (If 2 files are compared it is pretty trivial to notice a common structure).
:PROPERTIES:
:CUSTOM_ID: while-looking-through-the-file-if-2-files-are-compared-it-is-pretty-trivial-to-notice-a-common-structure.
:END:
#+begin_src go
// --------- Example ------------
//export StartContainer
func StartContainer(IP string) (output *C.char) {
container, err := client.StartContainer(IP, 0, false, "", "")
if err != nil {
return C.CString(err.Error())
}
return ConvertStructToJSONString(container)
}
//export ViewPlugin
func ViewPlugin() (output *C.char) {
plugins, err := plugin.DetectPlugins()
if err != nil {
return C.CString(err.Error())
}
return ConvertStructToJSONString(plugins)
}
#+end_src
**** It is easy to notice that:
:PROPERTIES:
:CUSTOM_ID: it-is-easy-to-notice-that
:END:
- =ConvertStructToJSONString(<go object>)=: This is a helper function
that convert a go object to JSON string initially and converts it to
=CString=.
- =(output *C.char)=: This is the return type for most of the functions.
**** A Pseudo code to refer to the common function implementation shape could be represented as:
:PROPERTIES:
:CUSTOM_ID: a-pseudo-code-to-refer-to-the-common-function-implementation-shape-could-be-represented-as
:END:
#+begin_example
func <Function name> (output *C.char) {
<response>,<error> := <P2PRC function name>(<parameters if needed>)
if <error> != nil {
return C.CString(<error>.Error())
}
return ConvertStructToJSONString(<response>)
}
#+end_example
** Current languages supported
:PROPERTIES:
:CUSTOM_ID: current-languages-supported
:END:
- Python
*** Build sample python program
:PROPERTIES:
:CUSTOM_ID: build-sample-python-program
:END:
The easier way
#+begin_src sh
# Run
make python
# Expected ouput
Output is in the Directory Bindings/python/export/
# Run
cd Bindings/python/export/
# list files
ls
# Expected output
SharedObjects/ p2prc.py
#+end_src
Above shows a generated folder which consists of a folder called
"SharedObjects/" which consists of =p2prc.so= and =p2prc.h= files.
=p2prc.py= refers to a sample python script calling P2PRC go functions.
To start an any project to extend P2PRC with python, This generated
folder can copied and created as a new git repo for P2PRC extensions
scripted or used a reference point as proof of concept that P2PRC can be
called from other programming languages.

24
Docs/CliImplementation.md.org Normal file
View File

@@ -0,0 +1,24 @@
* Cli module
:PROPERTIES:
:CUSTOM_ID: cli-module
:END:
The Cli (i.e Command Line Interface) is the only one in which the user
can directly interact with the modules in the project. The objective
when building the Cli was to have the least amount of commands as
possible. The cli was built using the library called urfave cli v2 .
They were 2 major files created named as flags.go and actions.go. ###
Flags.go The flags .go file is responsible to create the appropriate
flags for the cli. There are 2 types of flags called boolean and string.
Each of the flags outputs are assigned to a variable to be handled. The
flags can also detect environment variables set. This feature is useful
because if the user wants to call certain flags in a repeated sequence
it only has to be initialized once.
*** Actions.go
:PROPERTIES:
:CUSTOM_ID: actions.go
:END:
The actions.go file is implemented to call the appropriate functions
when the flags are called. It interacts directly with the modules in the
project. Action.go checks if variables are not empty string or the
boolean value is true.

33
Docs/Client.md.org Normal file
View File

@@ -0,0 +1,33 @@
* Client Module
:PROPERTIES:
:CUSTOM_ID: client-module
:END:
This module is incharge of communicating with the server and receiving
the appropriate information back from the server.
** Functions of the Client Module
:PROPERTIES:
:CUSTOM_ID: functions-of-the-client-module
:END:
#+begin_html
<!-- - [Interact with the Server Api](#functions-of-the-client-module) -->
#+end_html
- [[#decision-maker-on-how-the-ip-table-is-created-or-updated][Decision
maker on how the ip table is created or updated]]
** Decision maker on how the IP table is created or updated
:PROPERTIES:
:CUSTOM_ID: decision-maker-on-how-the-ip-table-is-created-or-updated
:END:
- Does a local speedtest to verify and see if the server IP's in the IP
table are pingable.
- Tries to ping the servers IP Table addresses.
- If it's pingable then it's added as a new entry in the IP table.
- The following steps occurs in the clients IP table.
- To ensure that the same servers are not being called to update the IP
table. There is a temporary list of IP address which have already been
called in relation to updating the IP table.
- Based on the current implementation there will 3 hops done to update
the IP table.

15
Docs/ClientArchitecture.md.org Normal file
View File

@@ -0,0 +1,15 @@
* Client Module Architecture
:PROPERTIES:
:CUSTOM_ID: client-module-architecture
:END:
The Client Module interacts with the P2P module and Server Module. It is
responsible for interacting with the server module and appropriately
updating the IP table on the client side. It connects to the server
using the server's REST Apis. It is also the primary decision maker on
how the IP table is updated is on the client side. This is because each
user can have requirements like how many number of hops they would want
to do to update their IP table. Hops is the number of times the client
is going to download the IP table from different servers ,once it gets
the IP tables from the previous servers.
[[file:images/NumOfHops.png]] [[file:images/clientmoduleArch.png]]

135
Docs/ClientImplementation.md.org Normal file
View File

@@ -0,0 +1,135 @@
* Client Module Implementation
:PROPERTIES:
:CUSTOM_ID: client-module-implementation
:END:
The Client Module interacts with the P2P module and Server Module. It is
responsible for interacting with the server module and appropriately
updating the IP table on the client side. It connects to the server
using the server's REST Apis. It is also the primary decision maker on
how the IP table is updated is on the client side. This is because each
user can have requirements like how many number of hops they would want
to do to update their IP table. Hops is the number of times the client
is going to download the IP table from different servers ,once it gets
the IP tables from the previous servers.
[[file:images/NumOfHops.png]] [[file:images/clientmoduleArch.png]]
** Topics
:PROPERTIES:
:CUSTOM_ID: topics
:END:
1. [[#updating-the-IP-table][Updating the IP table]]
2. [[#reading-server-specifications][Reading server specifications]]
3. [[#Client-creating-and-removing-container][Client creating and
removing container]]
4. [[#Tracking-Containers][Tracking Containers]]
5. [[#Grouping-Containers][Grouping Containers]]
This section focuses in depth on how the client module works. The client
module is incharge of communicating with different servers based on the
IP addresses provided to the user. The IP addresses are derived from
peer to peer modules. The objective here is how the client module
interacts with peer to peer module and server module.
*** Updating the IP table
:PROPERTIES:
:CUSTOM_ID: updating-the-ip-table
:END:
The client module calls the peer to peer module to get the local IP
table initially, Based on the servers IP addresses available it calls
the speedtest function from the peer to peer module to update IP
addresses with information such as latencies, download and upload
speeds. Once this is done the client module does a Rest Api call to the
server to download its IP Table. Once the hops are done it writes the
appropriate results to the Local IP table. Once this is done it prints
out the results. To derive parameters such as current the public IP
address the url "http://ip-api.com/json/" was called. This url returns
json response of the current public IP address. This feature will be
used in the future to ensure that the user's current IP address will not
be used for a speed test. Clients IP table is updated to the server
using a form of type multipart.
*** Reading server specifications
:PROPERTIES:
:CUSTOM_ID: reading-server-specifications
:END:
The client module calls the route /server_specs and reads the json
response. If the json response was successful then it just calls the
pretty print function which just prints the json output in the terminal.
*** Client creating and removing container
:PROPERTIES:
:CUSTOM_ID: client-creating-and-removing-container
:END:
The client module uses the servers Rest apis to create and delete
containers. To create a container the client requires 3 parameters being
the server ip address, the number of the ports the user wants to open
and if the user wants it connected to the GPU or not. The 3 parameters
are sent as a GET request to the server and the server responds with a
json file which has information such as the container ID, ports open ,
SSH username, SSH password, VNC username and VNC password. At the moment
the username and password are hard coded from the server side for both
SSH and VNC. To remove a container the client module only requires the
server IP address and the container ID. The client prints the response
from the server Rest api.
*** Tracking Containers
:PROPERTIES:
:CUSTOM_ID: tracking-containers
:END:
Clients create docker images in multiple machines. This means if the
client (i.e user) has many containers created there needs to be a way to
track them. To track containers there is a file called
=trackcontainers.json= which tracks all the containers running. The
snippet below show a sample structure of file =trackcontainer.json=.
#+begin_example
{
"TrackContainer": [
{
"ID": "<ID>",
"Container": {<docker.DockerVM struct>},
"IpAddress": "<IP Address>"
}
]
}
#+end_example
The default path to the container tracker is
=client/trackcontainers/trackcontainers.json=.
*** Grouping Containers
:PROPERTIES:
:CUSTOM_ID: grouping-containers
:END:
When starting a set container possibility to be able to group them. The
benefit this would be that when executing plugins the group ID would be
enough to execute plugin in a set of containers. This provides the
possibility to execute repetitive tasks in containers in a single cli
command. To store groups there is a file called
=grouptrackcontainer.json= which tracks all the groups currently present
set by the client. The snippet below show a sample structure of file
=grouptrackcontainer.json=.
#+begin_example
{
"Groups": [
{
"ID": "grp<Random UUID>",
"TrackContainer": [{client.TrackContainers struct}]
}
]
}
#+end_example
The default path to the container tracker is
=client/trackcontainers/grouptrackcontainer.json=.
#+begin_quote
[!NOTE] The group id will be auto-generated and will have its own prefix
in the start which will mostly be =grp<UUID>=.\\
When a container is removed using the command.
=p2prc --rm <IP Address> --id <Container id>=. It will be automatically
deleted from the groups it exists in.
#+end_quote

35
Docs/ConfigImplementation.md.org Normal file
View File

@@ -0,0 +1,35 @@
* Config Implementation
:PROPERTIES:
:CUSTOM_ID: config-implementation
:END:
The configuration module is responsible to store basic information of
absolute paths of files being called in the Go code. In a full-fledged
Cli the configuration file can be found in the directory /etc/ and from
there points to location such as where the IP table file is located. In
the future implementation the config file will have information such as
number of hops and other parameters to tweak and to improve the
effectiveness of the peer to peer network. The configuration module was
implemented using the library Viper. The Viper library automates
features such as searching in default paths to find out if the
configuration file is present. If the configuration file is not present
in the default paths then it auto generates the configuration file. The
configurations file can be in any format. In this project the
configuration file was generated using JSON format.
#+begin_src json
{
"MachineName": "pc-74-120.customer.ask4.lan",
"IPTable": "/Users/akilan/Documents/p2p-rendering-computation/p2p/iptable/ip_table.json",
"DockerContainers": "/Users/akilan/Documents/p2p-rendering-computation/server/docker/containers/",
"DefaultDockerFile": "/Users/akilan/Documents/p2p-rendering-computation/server/docker/containers/docker-ubuntu-sshd/",
"SpeedTestFile": "/Users/akilan/Documents/p2p-rendering-computation/p2p/50.bin",
"IPV6Address": "",
"PluginPath": "/Users/akilan/Documents/p2p-rendering-computation/plugin/deploy",
"TrackContainersPath": "/Users/akilan/Documents/p2p-rendering-computation/client/trackcontainers/trackcontainers.json",
"ServerPort": "8088",
"GroupTrackContainersPath": "/Users/akilan/Documents/p2p-rendering-computation/client/trackcontainers/grouptrackcontainers.json",
"FRPServerPort": "True",
"BehindNAT": "True",
"CustomConfig": null
}
#+end_src

26
Docs/DesignArchtectureIntro.md.org Normal file
View File

@@ -0,0 +1,26 @@
* Design Architecture
:PROPERTIES:
:CUSTOM_ID: design-architecture
:END:
This chapter focuses on architecture of the dissertation. The objective
would be to have a good understanding on the purpose of each module and
how they interact with each other. The design architecture was inspired
and based on the linux kernel design. The project is segmented into
various modules. Each module is responsible for certain tasks in the
project. The modules are highly dependent on each other hence the entire
codebase can be considered as a huge monolithic chuck which acts as its
own library. The following sub topics below talk about the main modules
and how they function with appropriate diagrams.
*** 1. [[file:ClientArchitecture.md][Client Module]]
:PROPERTIES:
:CUSTOM_ID: client-module
:END:
*** 2. [[file:P2PArchitecture.md][P2P Module]]
:PROPERTIES:
:CUSTOM_ID: p2p-module
:END:
*** 3. [[file:ServerArchitecture.md][Server Module]]
:PROPERTIES:
:CUSTOM_ID: server-module
:END:

5
Docs/DomainNameMappingsImplementation.md.org Normal file
View File

@@ -0,0 +1,5 @@
* Domain name mappings
:PROPERTIES:
:CUSTOM_ID: domain-name-mappings
:END:
This

66
Docs/Implementation.md.org Normal file
View File

@@ -0,0 +1,66 @@
* Implementation
:PROPERTIES:
:CUSTOM_ID: implementation
:END:
| [[file:Introduction.md][◀ Previous]] | [[file:README.md][Back to TOC]] |
|--------------------------------------+---------------------------------|
This chapter describes how the project was built. It talks in depth of
the implementation performed to give a better understanding of the
project.
** Programming langauge used
:PROPERTIES:
:CUSTOM_ID: programming-langauge-used
:END:
The programming language used for this project was
[[https://go.dev/][Golang]]. The reason Go lang was chosen was because
it is a compiled language. The entire codebase is just a single binary
file. When distributing to other linux distributing the only requirement
would be the binary file to run the code. It is easy to write
independant modules and be monolithic at the sametime using Go. Using
Go.mod makes it very easy to handle external libraries and modularise
code. The go.mod name for the project is
[[https://git.sr.ht/~akilan1999/p2p-rendering-computation][git.sr.ht/~akilan1999/p2p-rendering-computation]].
-
** [[file:CliImplementation.md][Cli Module]]
:PROPERTIES:
:CUSTOM_ID: cli-module
:END:
-
** [[file:ConfigImplementation.md][Config Module]]
:PROPERTIES:
:CUSTOM_ID: config-module
:END:
-
** [[file:ServerImplementation.md][Server Module]]
:PROPERTIES:
:CUSTOM_ID: server-module
:END:
-
** [[file:ClientImplementation.md][Client Module]]
:PROPERTIES:
:CUSTOM_ID: client-module
:END:
-
** [[file:P2PImplementation.md][P2P Module]]
:PROPERTIES:
:CUSTOM_ID: p2p-module
:END:
-
** [[file:PluginImplementation.md][Plugin Module]]
:PROPERTIES:
:CUSTOM_ID: plugin-module
:END:
-
** [[file:GenerateImplementation.md][Generate Module]]
:PROPERTIES:
:CUSTOM_ID: generate-module
:END:

361
Docs/Installation.md.org Normal file
View File

@@ -0,0 +1,361 @@
* Installation
:PROPERTIES:
:CUSTOM_ID: installation
:END:
| [[file:Introduction.md][◀ Previous]] | [[file:Abstractions.md][Next ▶]] |
|--------------------------------------+----------------------------------|
Over here we will cover the basic steps to get the server and client
side running.
** Latest release install
:PROPERTIES:
:CUSTOM_ID: latest-release-install
:END:
https://github.com/Akilan1999/p2p-rendering-computation/releases
** Install from Github master branch
:PROPERTIES:
:CUSTOM_ID: install-from-github-master-branch
:END:
*** Install Go lang
:PROPERTIES:
:CUSTOM_ID: install-go-lang
:END:
The entire the implementation of this project is done using Go lang.
Thus, we need go lang to compile to code to a binary file.
[[https://golang.org/doc/install][Instructions to install Go lang]]
*** Install Docker
:PROPERTIES:
:CUSTOM_ID: install-docker
:END:
In this project the choice of virtualization is Docker due to it's wide
usage in the developer community. In the server module we use the Docker
Go API to create and interact with the containers.
[[https://docs.docker.com/get-docker/][Instructions to install docker]]
[[https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker][Instructions
to install docker GPU]]
#+begin_example
// Do ensure that the docker command does not need sudo to run
sudo chmod 666 /var/run/docker.sock
#+end_example
*** Build Project and install project
:PROPERTIES:
:CUSTOM_ID: build-project-and-install-project
:END:
To set up the internal dependencies and build the entire go code into a
single binary
#+begin_example
make install
#+end_example
**** For Windows
:PROPERTIES:
:CUSTOM_ID: for-windows
:END:
To set up P2PRC on Windows, simply run this batch file. *Make sure you
are not in admin mode when running this.*
#+begin_example
.\install.bat
#+end_example
*** Add appropriate paths to =.bashrc=
:PROPERTIES:
:CUSTOM_ID: add-appropriate-paths-to-.bashrc
:END:
#+begin_example
export P2PRC=/<PATH>/p2p-rendering-computation
export PATH=/<PATH>/p2p-rendering-computation:${PATH}
#+end_example
*** Set up configuration file
:PROPERTIES:
:CUSTOM_ID: set-up-configuration-file
:END:
#+begin_example
make configfile
#+end_example
Open the config file =config.json= and add the IPv6 address if you have
one.
*** Test if binary works
:PROPERTIES:
:CUSTOM_ID: test-if-binary-works
:END:
#+begin_example
p2prc --help
#+end_example
**** Output:
:PROPERTIES:
:CUSTOM_ID: output
:END:
#+begin_example
NAME:
p2p-rendering-computation - p2p cli application to create and access VMs in other servers
USAGE:
p2prc [global options] command [command options] [arguments...]
VERSION:
<version no>
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--Server, -s Starts server (default: false) [$SERVER]
--UpdateServerList, --us Update List of Server available based on servers iptables (default: false) [$UPDATE_SERVER_LIST]
--ListServers, --ls List servers which can render tasks (default: false) [$LIST_SERVERS]
--AddServer value, --as value Adds server IP address to iptables [$ADD_SERVER]
--ViewImages value, --vi value View images available on the server IP address [$VIEW_IMAGES]
--CreateVM value, --touch value Creates Docker container on the selected server [$CREATE_VM]
--ContainerName value, --cn value Specifying the container run on the server side [$CONTAINER_NAME]
--RemoveVM value, --rm value Stop and Remove Docker container (IP:port) accompanied by container ID via --ID or --id [$REMOVE_VM]
--ID value, --id value Docker Container ID [$ID]
--Ports value, -p value Number of ports to open for the Docker Container [$NUM_PORTS]
--GPU, --gpu Create Docker Containers to access GPU (default: false) [$USE_GPU]
--Specification value, --specs value Specs of the server node [$SPECS]
--SetDefaultConfig, --dc Sets a default configuration file (default: false) [$SET_DEFAULT_CONFIG]
--NetworkInterfaces, --ni Shows the network interface in your computer (default: false) [$NETWORK_INTERFACE]
--ViewPlugins, --vp Shows plugins available to be executed (default: false) [$VIEW_PLUGIN]
--TrackedContainers, --tc View (currently running) containers which have been created from the client side (default: false) [$TRACKED_CONTAINERS]
--ExecutePlugin value, --plugin value Plugin which needs to be executed [$EXECUTE_PLUGIN]
--CreateGroup, --cgroup Creates a new group (default: false) [$CREATE_GROUP]
--Group value, --group value group flag with argument group ID [$GROUP]
--Groups, --groups View all groups (default: false) [$GROUPS]
--RemoveContainerGroup, --rmcgroup Remove specific container in the group (default: false) [$REMOVE_CONTAINER_GROUP]
--RemoveGroup value, --rmgroup value Removes the entire group [$REMOVE_GROUP]
--Generate value, --gen value Generates a new copy of P2PRC which can be modified based on your needs [$GENERATE]
--ModuleName value, --mod value New go project module name [$MODULENAME]
--PullPlugin value, --pp value Pulls plugin from git repos [$PULLPLUGIN]
--RemovePlugin value, --rp value Removes plugin [$REMOVEPLUGIN]
--help, -h show help (default: false)
--version, -v print the version (default: false)
#+end_example
--------------
* Using basic commands
:PROPERTIES:
:CUSTOM_ID: using-basic-commands
:END:
*** Start as a server
:PROPERTIES:
:CUSTOM_ID: start-as-a-server
:END:
Do ensure you have Docker installed for this
#+begin_example
p2prc -s
#+end_example
*** View server Specification
:PROPERTIES:
:CUSTOM_ID: view-server-specification
:END:
#+begin_example
p2prc --specs=<ip address>
#+end_example
*** Run container
:PROPERTIES:
:CUSTOM_ID: run-container
:END:
use the =--gpu= if you know the other machine has a gpu.
#+begin_example
p2prc --touch=<server ip address> -p <number of ports> --gpu
#+end_example
*** Remove container
:PROPERTIES:
:CUSTOM_ID: remove-container
:END:
The docker id is present in the output where you create a container
#+begin_example
p2prc --rm=<server ip address> --id=<docker container id>
#+end_example
*** Adding servers to ip table
:PROPERTIES:
:CUSTOM_ID: adding-servers-to-ip-table
:END:
#+begin_example
p2prc --as=<server ip address you want to add>
#+end_example
*** Update ip table
:PROPERTIES:
:CUSTOM_ID: update-ip-table
:END:
#+begin_example
p2prc --us
#+end_example
*** List Servers
:PROPERTIES:
:CUSTOM_ID: list-servers
:END:
#+begin_example
p2prc --ls
#+end_example
*** View Network interfaces
:PROPERTIES:
:CUSTOM_ID: view-network-interfaces
:END:
#+begin_example
p2prc --ni
#+end_example
*** Viewing Containers created Client side
:PROPERTIES:
:CUSTOM_ID: viewing-containers-created-client-side
:END:
#+begin_example
p2prc --tc
#+end_example
[[file:ClientImplementation.md#tracking-containers][read more on
tracking containers]]
*** Running plugin
:PROPERTIES:
:CUSTOM_ID: running-plugin
:END:
#+begin_example
p2prc --plugin <plugin name> --id <container id or group id>
#+end_example
*** Create group
:PROPERTIES:
:CUSTOM_ID: create-group
:END:
#+begin_example
p2prc --cgroup
#+end_example
*** Add container to group
:PROPERTIES:
:CUSTOM_ID: add-container-to-group
:END:
#+begin_example
p2prc --group <group id> --id <container id>
#+end_example
*** View groups
:PROPERTIES:
:CUSTOM_ID: view-groups
:END:
#+begin_example
p2prc --groups
#+end_example
*** View specific group
:PROPERTIES:
:CUSTOM_ID: view-specific-group
:END:
#+begin_example
p2prc --group <group id>
#+end_example
*** Delete container from group
:PROPERTIES:
:CUSTOM_ID: delete-container-from-group
:END:
#+begin_example
p2prc --rmcgroup --group <group id> --id <container id>
#+end_example
*** Delete entire group
:PROPERTIES:
:CUSTOM_ID: delete-entire-group
:END:
#+begin_example
p2prc --rmgroup <group id>
#+end_example
[[file:ClientImplementation.md#Grouping-Containers][read more on
grouping containers]] ### Extending usecase of P2PRC (Requires a go
compiler to run)
#+begin_example
p2prc --gen <project name> --mod <go module name>
#+end_example
[[file:GenerateImplementation.md][read more about the generate module]]
*** Pulling plugin from a remote repo
:PROPERTIES:
:CUSTOM_ID: pulling-plugin-from-a-remote-repo
:END:
#+begin_example
p2prc --pp <repo link>
#+end_example
*** Deleting plugin from the plugin directory
:PROPERTIES:
:CUSTOM_ID: deleting-plugin-from-the-plugin-directory
:END:
#+begin_example
p2prc --rp <plugin name>
#+end_example
*** Added custom metadata about the current node
:PROPERTIES:
:CUSTOM_ID: added-custom-metadata-about-the-current-node
:END:
#+begin_example
p2prc --amd "custom metadata"
#+end_example
--------------
* Using Plugins
:PROPERTIES:
:CUSTOM_ID: using-plugins
:END:
This feature is still Under Development:
[[file:PluginImplementation.md][Read more on the implementation]]
**** Dependencies
:PROPERTIES:
:CUSTOM_ID: dependencies
:END:
- Ansible:
- Debian/ubuntu: =sudo apt install ansible=
- Others:
[[https://ansible-tips-and-tricks.readthedocs.io/en/latest/ansible/install/][Installation
link]]
**** Run Test Cases
:PROPERTIES:
:CUSTOM_ID: run-test-cases
:END:
- Generate Test Case Ansible file
- =make testcases=
- Enter inside plugin directory and run tests.
#+begin_quote
[!NOTE] That docker needs to installed and needs to run without sudo.
Refer the section [[#install-docker][Install Docker]]. - =cd plugin= -
=go test .=
#+end_quote
--------------
*** Next Chapter: [[file:Abstractions.md][Abstractions]]
:PROPERTIES:
:CUSTOM_ID: next-chapter-abstractions
:END:

48
Docs/Introduction.md.org Normal file
View File

@@ -0,0 +1,48 @@
* Chapter 1: Introduction
:PROPERTIES:
:CUSTOM_ID: chapter-1-introduction
:END:
| [[file:README.md][◀ Back to TOC]] | [[file:Installation.md][Next ▶]] |
|-----------------------------------+----------------------------------|
** Abstract
:PROPERTIES:
:CUSTOM_ID: abstract
:END:
This project focuses on creating a framework on running heavy tasks that
a regular computer cannot run easily such as graphically demanding video
games, rendering 3D animations , protein folding simulations. In this
project the major focus will not be on the financial incentive part. A
peer to peer network will be created to help run tasks decentrally,
increasing bandwidth for running tasks. To ensure the tasks in the peer
to peer network do not corrupt the server 0S (Operating System), they
will be executed in a virtual environment in the server.
The main aim of this project was to create a custom peer to peer
network. The user acting as the client has total flexibility on how to
batch the tasks and the user acting as the server has complete
flexibility on tracking the container's usages and killing the
containers at any point of time.
** Motivation
:PROPERTIES:
:CUSTOM_ID: motivation
:END:
Many of the users rely on our PC / Laptop or servers that belong to a
server farm to run heavy tasks and with the demand of high creativity
requires higher computing power. Buying a powerful computer every few
years to run a bunch of heavy tasks which are not executed as frequently
to reap the benefits can be inefficient utilization of hardware. On the
other end, renting servers to run these heavy tasks can be really
useful. Ethically speaking this is leading to monopolisation of
computing power similar to what is happening in the web server area. By
using peer to peer principles it is possible to remove the
monopolisation factor and increase the bandwidth between the client and
server.
--------------
*** Next Chapter: [[file:Installation.md][Installation]]
:PROPERTIES:
:CUSTOM_ID: next-chapter-installation
:END:

50
Docs/NAT-Traveral.md.org Normal file
View File

@@ -0,0 +1,50 @@
* NAT Traversal
:PROPERTIES:
:CUSTOM_ID: nat-traversal
:END:
P2PRC currently supports TURN for NAT traversal.
** TURN
:PROPERTIES:
:CUSTOM_ID: turn
:END:
The current TURN implementation used is FRP. The TURN server is also
required when a P2PRC node is acting as a Server. The TURN server is
determined based on the Node with the least amount of latency based on
the Nodes available on the IPTable. Once a TURN server is determined
there are 2 actions performed. The first one is =/FRPPort= to the TURN
server to receive a port which is used to generate the external port
from the TURN server. The flow below describes the workflow.
*** Client mode
:PROPERTIES:
:CUSTOM_ID: client-mode
:END:
- Call =/FRPPort=
#+begin_example
http://<turn server ip>:<server port no>/FRPport
#+end_example
- Call the TURN server in the following manner. The following is a
sample code snippet below.
#+begin_src go
import (
"github.com/Akilan1999/p2p-rendering-computation/p2p/frp"
)
func main() {
serverPort, err := frp.GetFRPServerPort("http://" + <lowestLatencyIpAddress.Ipv4> + ":" + lowestLatencyIpAddress.ServerPort)
if err != nil {
return nil, err
}
// Create 1 second delay to allow FRP server to start
time.Sleep(1 * time.Second)
// Starts FRP as a client with
proxyPort, err := frp.StartFRPClientForServer(<lowestLatencyIpAddress.Ipv4>, serverPort, <the port you want to expose externally>)
if err != nil {
return nil, err
}
}
#+end_src

136
Docs/P2P-testing.md.org Normal file
View File

@@ -0,0 +1,136 @@
* Testing P2P network
:PROPERTIES:
:CUSTOM_ID: testing-p2p-network
:END:
The objective would be to test the p2p network, and the effectiveness of
updating the ip tables. The objective of would be to give the impression
to the client and server of a Zero configuration setting. For testing
there will be a test network set. In the testing scenario all will be
client and server because the IP table does not store clients IP
addresses. At current number of hopes would be 3 as default.
*** Test Network Scenario 1
:PROPERTIES:
:CUSTOM_ID: test-network-scenario-1
:END:
The test network consists of 5 nodes acting as a client and server. The
objective would be to have the entire IP table Updated in each node with
interacting with only 1 node once. Each node has knowledge of one node
only.
[[https://user-images.githubusercontent.com/31743758/115069627-e4aa8c80-9f04-11eb-8402-706a3407f0e8.png]]
Fig 1.0 Visual Representation of testnet scenario 1
**** Result
:PROPERTIES:
:CUSTOM_ID: result
:END:
All nodes except node 1 where able to have information of IP addresses
in the test net. This was due to the reason of 3 hops set as default.
Node 1 had in it's IP table IP addresses of Node 2, Node 3, Node 4. Once
the number of hops was set to 4 objective of the test was acheived.
*** Test Network Scenario 2
:PROPERTIES:
:CUSTOM_ID: test-network-scenario-2
:END:
The second test network has a scenario of a single peer which all the
other nodes connect too. The scenario being when the other nodes connect
to the single server they download information about nodes that have
connected to the server node before.
*** Testing Broadcast Module
:PROPERTIES:
:CUSTOM_ID: testing-broadcast-module
:END:
For testing the broadcast module 2 types of servers will be tested. One
with a CPU only , another one with a CPU and GPU. The expected result
being that the appropriate results are visible.
**** Results (CPU and GPU):
:PROPERTIES:
:CUSTOM_ID: results-cpu-and-gpu
:END:
#+begin_example
{
"Hostname": "akilan-Lenovo-IdeaPad-Y510P",
"Platform": "ubuntu",
"CPU": "Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz",
"RAM": 7872,
"Disk": 937367,
"GPU": {
"DriveVersion": "390.141",
"Gpu": {
"GpuName": "GeForce GT 755M",
"BiosVersion": "80.07.A8.00.0F",
"FanSpeed": "N/A",
"Utilization": {
"GpuUsage": "N/A",
"MemoryUsage": "N/A"
},
"Temperature": {
"GpuTemp": "66 C"
},
"Clock": {
"GpuClock": "N/A",
"GpuMemClock": "N/A"
}
}
}
}
#+end_example
At the moment of the current implementation v1.0. Nvidia GPU are only
compatible. As the Go code calls the command =nvidia-smi= to get
information about the GPU available.
**** Results (CPU only)
:PROPERTIES:
:CUSTOM_ID: results-cpu-only
:END:
#+begin_example
{
"Hostname": "sv-t1.small.x86-01",
"Platform": "ubuntu",
"CPU": "Intel(R) Atom(TM) CPU C2750 @ 2.40GHz",
"RAM": 7944,
"Disk": 138793,
"GPU": null
}
#+end_example
As the =nvidia-smi= interface was not detected it only broadcasts the
CPU specs available.
*** SpeedTests
:PROPERTIES:
:CUSTOM_ID: speedtests
:END:
The speed test has 3 parameters which are Ping , upload and download.
The tests check if the results returned are approximately correct. The
ping at the moment returns the correct result. The upload and download
returned are inccorect at the moment, This is due incorrect
implementation in for timer and will be patched in future versions.
*** Unit tests
:PROPERTIES:
:CUSTOM_ID: unit-tests
:END:
All functions implemented on the P2P module returns type error. The
units test call certain functions and check if the functions return an
error or not. This proved sufficient as the point of the units tests was
code coverage to check if certain functions return an error.
**** Functions tested
:PROPERTIES:
:CUSTOM_ID: functions-tested
:END:
This sections talks about the function called and represents code
coverage.
1. =TestServer_SpeedTest=: Function called LocalSpeedTestIpTable()
2. =TestReadIpTable=: Function called ReadIpTable()
The P2P module has a 100% code coverage in unit tests as both the unit
tests call directly or call within the function all the functions used
in the P2P module.

45
Docs/P2P.md.org Normal file
View File

@@ -0,0 +1,45 @@
* P2P (Peer to Peer module)
:PROPERTIES:
:CUSTOM_ID: p2p-peer-to-peer-module
:END:
In this repository the P2P module has been designed from sratch at the
point of this implementation.
[[https://pkg.go.dev/git.sr.ht/~akilan1999/p2p-rendering-computation@v0.0.0-20210404191839-6a046babcb02/p2p][More
about function implementation]]
** Terminology
:PROPERTIES:
:CUSTOM_ID: terminology
:END:
1. IPTable: Refers to a json file which stores information about the
current servers avaliable with the speedtest results ran from the
Node that triggered it.
#+begin_example
{
"ip_address": [
{
"ipv4": "localhost",
"latency": 14981051,
"download": 8142.122540206258,
"upload": 3578.766512629995,
}
]
}
#+end_example
** Responsibility
:PROPERTIES:
:CUSTOM_ID: responsibility
:END:
- To ensure the IP table has nodes which are pingable
- Taking to nodes behind NAT. [[file:NAT-Traversal][More about the
implementation]]...
#+begin_quote
[!NOTE] If you are running in server mode it is recommended to use
[[https://routerguide.net/when-and-how-to-setup-dmz-host-for-home-use/][DMZ]]
to bypass the
[[https://en.wikipedia.org/wiki/Network_address_translation][NAT]].
#+end_quote

16
Docs/P2PArchitecture.md.org Normal file
View File

@@ -0,0 +1,16 @@
* P2P Module Architecture
:PROPERTIES:
:CUSTOM_ID: p2p-module-architecture
:END:
The P2P module (i.e Peer to Peer Module) is responsible for storing the
IP table and interacting with the IP table. In the following
implementation of the P2P module ,the IP table stores information about
servers available in the network. The other functionality the P2P module
takes care of is doing the appropriate speed tests to the servers in the
IP table. This is for informing the users about nodes which are close by
and nodes which have quicker uploads and downloads speeds. The module is
responsible to ensure that there are no duplicate server IPs in the IP
table and to remove all server IPs which are not pingable.
#+caption: UML diagram of P2P module
[[file:images/p2pmoduleArch.png]]

111
Docs/P2PImplementation.md.org Normal file
View File

@@ -0,0 +1,111 @@
* P2P Module Implementation
:PROPERTIES:
:CUSTOM_ID: p2p-module-implementation
:END:
The P2P module (i.e Peer to Peer Module) is responsible for storing the
IP table and interacting with the IP table. In the following
implementation of the P2P module ,the IP table stores information about
servers available in the network. The other functionality the P2P module
takes care of is doing the appropriate speed tests to the servers in the
IP table. This is for informing the users about nodes which are close by
and nodes which have quicker uploads and downloads speeds. The module is
responsible to ensure that there are no duplicate server IPs in the IP
table and to remove all server IPs which are not pingable.
#+caption: UML diagram of P2P module
[[file:images/p2pmoduleArch.png]]
The peer to peer implementation was built from scratch. This is because
other peer to peer libraries were on the implementation of the
Distributed hash table. At the current moment all those heavy features
are not needed because the objective is to search and list all possible
servers available. The limitation being that to be a part of the network
the user has to know at least 1 server. The advantage of building from
scratch makes the module super light and possibility for custom
functions and structs. The sub topics below will mention the
implementations of each functionality in depth.
** IP Table
:PROPERTIES:
:CUSTOM_ID: ip-table
:END:
The ip table file is a json as the format with a list of servers ip
addresses, latencies, downloads and uploads speeds. The functions
implemented include read file, write file and remove duplicate IP
addresses. The remove duplicate IP address function exists because
sometimes servers IP tables can have the same ip addresses as what the
client has. The path of the IP table json file is received from the
configuration module.
#+begin_src json
{
"ip_address": [
{
"ipv4": "<ipv4 address>",
"latency": "<latency>",
"download": "<download>",
"upload": "<upload>"
"port no": "<server port no>",
}
]
}
#+end_src
*** Latency
:PROPERTIES:
:CUSTOM_ID: latency
:END:
The latency is measured in milliseconds. The route /server_info is
called from the server and time it takes to provide a json response is
recorded.
** NAT Traversal
:PROPERTIES:
:CUSTOM_ID: nat-traversal
:END:
P2PRC currently supports TURN for NAT traversal.
** TURN
:PROPERTIES:
:CUSTOM_ID: turn
:END:
The current TURN implementation used is FRP. The TURN server is also
required when a P2PRC node is acting as a Server. The TURN server is
determined based on the Node with the least amount of latency based on
the Nodes available on the IPTable. Once a TURN server is determined
there are 2 actions performed. The first one is =/FRPPort= to the TURN
server to receive a port which is used to generate the external port
from the TURN server. The flow below describes the workflow.
*** Client mode
:PROPERTIES:
:CUSTOM_ID: client-mode
:END:
- Call =/FRPPort=
#+begin_example
http://<turn server ip>:<server port no>/FRPport
#+end_example
- Call the TURN server in the following manner. The following is a
sample code snippet below.
#+begin_src go
import (
"github.com/Akilan1999/p2p-rendering-computation/p2p/frp"
)
func main() {
serverPort, err := frp.GetFRPServerPort("http://" + <lowestLatencyIpAddress.Ipv4> + ":" + lowestLatencyIpAddress.ServerPort)
if err != nil {
return nil, err
}
// Create 1 second delay to allow FRP server to start
time.Sleep(1 * time.Second)
// Starts FRP as a client with
proxyPort, err := frp.StartFRPClientForServer(<lowestLatencyIpAddress.Ipv4>, serverPort, <the port you want to expose externally>)
if err != nil {
return nil, err
}
}
#+end_src

213
Docs/PluginImplementation.md.org Normal file
View File

@@ -0,0 +1,213 @@
* Plugin Module Implementation
:PROPERTIES:
:CUSTOM_ID: plugin-module-implementation
:END:
** Topics
:PROPERTIES:
:CUSTOM_ID: topics
:END:
1. [[#introduction][Introduciton]]
2. [[#site-File-Template][Site.yml]]
3. [[#hosts-file][Host]]
4. [[#description-file][Description]]
5. [[#automatic-port-allocations][Automatic port allocations]]
6. [[#sample-plugins-implemented][Sample plugins implemented]]
** Introduction
:PROPERTIES:
:CUSTOM_ID: introduction
:END:
The plugin module is designed to ensure clients can execute instructions
in a declarative manner across different containers created. This means
the user (i.e client) needs to write the instruction only once, and
these instructions can be executed across different nodes in a
repetitive manner.
In the scenario of this project Ansibles will be used as the way the
users can create these instructions.
- [[file:Installation.md#Using-Plugins][Setup instruction]]
The plugin module introduces a new path to the config file known as
pluginpath. This path by defaults points to =${P2PRC}/plugin/deploy=.
Any file/folder inside =plugin/deploy= is part of the .gitginore.
Plugins are detected by folder names inside the =plugin/deploy=.
#+begin_example
plugin
|___ Deploy
|___<plugin name>
|___ site.yml
|___ hosts
|___ ports.json
|___ description.txt
.
.
.
n: n number of plugins possible
#+end_example
** Site File Template
:PROPERTIES:
:CUSTOM_ID: site-file-template
:END:
The site file is also known as the Ansible playbook and is incharge of
executing instructions in a declarative manner. The below example
specifies how to make one.
#+begin_example
- hosts: all
tasks:
- name: <task name>
<ansible task>
debug:
msg: <debug message>
#+end_example
Read more about ansible tasks:
https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html#about-playbooks
** Hosts file
:PROPERTIES:
:CUSTOM_ID: hosts-file
:END:
hosts file is also known as the inventory file. This file consists of
all the information required to connect to other nodes to execute
Ansible instructions. In this project this file needs to be set in a
certain configuration because the go code or binary will populate this
file automatically with the appropriate information required to connect
to local or remote containers.
#+begin_quote
[!NOTE] Add as exactly specified below:
#+begin_example
all:
vars:
ansible_python_interpreter: /usr/bin/python3 // Path to your python 3 interpreter
main:
hosts:
host1:
// Note: These values will be automatically overwritten
// by the Go functions
ansible_host: 0.0.0.0
ansible_port: 39269
ansible_user: master
ansible_ssh_pass: password
ansible_sudo_pass: password
#+end_example
#+end_quote
** Ports.json
:PROPERTIES:
:CUSTOM_ID: ports.json
:END:
The =ports.json= file is intended to mention the number of ports
required by the plugin.
#+begin_example
{
"NumOfPorts": <number of ports>
}
#+end_example
** Description file
:PROPERTIES:
:CUSTOM_ID: description-file
:END:
This is a simple text file used to describe what the module does. When
the client is looking at various commands via the ClI. The description
is displayed along-side the plugin name.
Ex: When the flag =--ViewPlugins= or =--vp= is called
#+begin_example
{
"PluginsDetected": [
{
"FolderName": "<name of the plugin>",
"PluginDescription": "<description of the plugin>"
}
]
}
#+end_example
** Automatic port allocations
:PROPERTIES:
:CUSTOM_ID: automatic-port-allocations
:END:
P2PRC would be in-charge to set to the ports to various TCP ports
opened. Due to this implementation the plugin being executed is copied
to the tmp directory with a unique UUID.
#+begin_example
Command: ls /tmp
output: Semantic <UUID>_<Plugin Name>
2e6d76c4-0ed1-4b55-9385-79a58d4f0492_p2prc-vscode-browser
7b631e08-62ee-4c1c-a2a4-c05857b9aa7d_p2prc-vscode-browser
#+end_example
Once the copy of the plugin is added to the /tmp directory the site.yml
file inside the appropriate yaml is modified with the appropriate ports
assigned to the container.
*** Ex:
:PROPERTIES:
:CUSTOM_ID: ex
:END:
1. Create container called c1 with an automatic generated TCP port 3313
(external) - 3313 (internal)
2. Assumption of plugin p1 exists. p1 has one server which needs to be
mapped to a free open TCP port in container c1. Below shows an
implementation of a sample site.yml file.
#+begin_example
---
- hosts: all
tasks:
- name: start vscode code server
shell: sh server.sh 0.0.0.0:{{index . 0}}
#+end_example
Notice there is the following {{index . 0}}. {{index . 0}} does not
belong to Ansible but rather is a way to mention where to add the
external free port of the container. We use the golang
[[https://pkg.go.dev/text/template][template library]] to parse and
populate the site.yml with the appropriate open ports. An array of ints
which consists of open free ports are sent to the site.yml. 0 in
{{index . 0}} refers to the index in the int array passed on.
After the port is automatically it's ready to run !
#+begin_example
---
- hosts: all
tasks:
- name: start vscode code server
shell: sh server.sh 0.0.0.0:3313
#+end_example
*** Sample plugins implemented:
:PROPERTIES:
:CUSTOM_ID: sample-plugins-implemented
:END:
- [[https://github.com/Akilan1999/p2prc-vscode-browser][VSCode Plugin]]
** Pull Plugins
:PROPERTIES:
:CUSTOM_ID: pull-plugins
:END:
The following allows us to pull plugins from a remote git repository and
store them in the default plugins directory. The implementation uses a
Go git library to pull the git repo and automatically save it as a
folder in the plugin path.
** Delete Plugins
:PROPERTIES:
:CUSTOM_ID: delete-plugins
:END:
We delete the plugin folder based on the plugin name provided as an
argument on the cli command. Once the folder is deleted, the plugin
manager automatically knows that the plugin does not exist anymore.

18
Docs/README.md.org Normal file
View File

@@ -0,0 +1,18 @@
* Table of contents
:PROPERTIES:
:CUSTOM_ID: table-of-contents
:END:
1. [[file:Introduction.md][Introduction]]
2. [[file:Installation.md][Installation]]
3. [[file:Abstractions.md][Abstractions]]
4. [[file:Implementation.md][Implementation]]
1. [[file:ClientImplementation.md][Client Module]]
2. [[file:P2PImplementation.md][P2P Module]]
3. [[file:ServerImplementation.md][Server Module]]
4. [[file:ConfigImplementation.md][Config Module]]
5. [[file:CliImplementation.md][Cli Module]]
6. [[file:PluginImplementation.md][Plugin Module]]
7. [[file:Bindings.md][Language bindings]]
8. [[file:Bindings.md][Domain name mappings]]
5. Language bindings
1. [[file:haskell/][Haskell]]

14
Docs/ServerArchitecture.md.org Normal file
View File

@@ -0,0 +1,14 @@
* Server Module Architecture
:PROPERTIES:
:CUSTOM_ID: server-module-architecture
:END:
The server module takes care of setting and removing the virtualization
environment (i.e containers) for accessing and doing the appropriate
computation. It also interacts with the peer to peer module to update
the IP table on the server side. The server module accesses information
regarding CPU and GPU specifications of the machine running the server
module. To do Speed tests the server has routes which allows it to
upload and download a 50mb.
#+caption: UML diagram of server module
[[file:images/servermoduleArch.png]]

278
Docs/ServerImplementation.md.org Normal file
View File

@@ -0,0 +1,278 @@
* Server Module Implementation
:PROPERTIES:
:CUSTOM_ID: server-module-implementation
:END:
This section focuses on an in-depth understanding of the server module
implementation. To understand the architecture of the server module
refer. The server module can be split into various sections. Each
section will provide information on how a certain feature works.
The server module takes care of setting and removing the virtualization
environment (i.e containers) for accessing and doing the appropriate
computation. It also interacts with the peer to peer module to update
the IP table on the server side. The server module accesses information
regarding CPU and GPU specifications of the machine running the server
module. To do Speed tests the server has routes which allows it to
upload and download a 50mb.
#+caption: UML diagram of server module
[[file:images/servermoduleArch.png]]
** Web framework
:PROPERTIES:
:CUSTOM_ID: web-framework
:END:
The web framework used for the server module is called Gin. The reason
Gin was chosen is due to its wide use and strong documentation available
on the official github repository. The default port used is 8088. For
version 1.0 of the project ,the server needs to keep port 8088 open to
ensure that other clients and servers can detect it. The possible
requests available are GET and POST for this implementation. The
possible responses are either a string or json response or a file. In
the majority of routes a string response refers to an error when calling
the following routes. The following sub topics below will talk about the
route implemented:
*** /server_info
:PROPERTIES:
:CUSTOM_ID: server_info
:END:
This route is responsible to get information about the specifications of
the server. The response of this route is in json if the call was
successful.
*** /50
:PROPERTIES:
:CUSTOM_ID: section
:END:
This route is responsible for returning a randomly generated 50mb file.
This is used to calculate the download speed from the p2p module.
*** /IpTable
:PROPERTIES:
:CUSTOM_ID: iptable
:END:
This route is a POST request that is responsible to update the server IP
table based on the IP table the client provides. Once the server gets
the IP table it checks if the client is also a server. This is done by
calling the url http://:8088/server_info. If the server_info route from
the client responds back with computer specifications of the client.
Then the server initially appends the clients IP to the struct. After
that the IP table received from the client is uploaded to the struct.
Once this is done the server passes the struct to the peer to peer
module function. The peer to peer module function will return the back
with the new struct with the valid server nodes. The server responds
back to the new struct as a json format. If a string is present in the
response then there is probably an error on the server side.
*** /startcontainer
:PROPERTIES:
:CUSTOM_ID: startcontainer
:END:
This route takes in a GET request with the number of TCP ports to open
and checks whether the docker container should be hooked to the GPU or
not. This route talks to the docker module implemented as a sub module
in the server module. More information on the docker module in section
5.4.3. This route calls docker the module to start the container for the
client. The docker module returns back a struct. This struct is returned
back to the client as the json response. This struct consists of
information such as docker id, ports numbers open , information
regarding SSH and VNC connections to the docker container created when
the client created this request.
*** /RemoveContainer
:PROPERTIES:
:CUSTOM_ID: removecontainer
:END:
This route takes in a GET request as the container ID. Based on the
container ID provided ,it calls the docker module which deletes the
container. If the deletion is successful it returns back a string which
says success.
** Server information/ Specification
:PROPERTIES:
:CUSTOM_ID: server-information-specification
:END:
This section provides information on how the server specifications are
read. There are 2 major implementations. The first implementation
mentions how basic information such as RAM usage, CPU specification are
detected and the second implementation mentions how the GPU drivers are
detected and information is extracted. The client has to assume that the
server is using default docker settings in terms of CPU cycles and other
parameters.
*** Basic Information
:PROPERTIES:
:CUSTOM_ID: basic-information
:END:
The file name for these functions is called gopsutil.go. This codebase
uses the library gopsutil. Gopsutil has various packages or modules
within the library which have functions implemented to get system
information. The following information is stored in a struct and the
function returns that struct.
#+begin_src go
type SysInfo struct {
Hostname string `bson:hostname`
Platform string `bson:platform`
CPU string `bson:cpu`
RAM uint64 `bson:ram`
Disk uint64 `bson:disk`
GPU *Query `xml: GpuInfo`
}
#+end_src
*** GPU Information
:PROPERTIES:
:CUSTOM_ID: gpu-information
:END:
The file name for these functions is called GPU.go. This codebase checks
if the Nvidia driver exists and returns the driver information. To do
this a shell command called nvidia-smi is executed. This shell command
is executed with a --xml as flag to ensure that the output is in the XML
format. If there is an output as a xml format, that means there is an
nvidia driver installed, and the function just reads the output and
stores it to the struct and returns the GPU information.
#+begin_src go
type Query struct {
DriveVersion string `xml:"driver_version"`
Gpu Gpu `xml:"gpu"`
}
type Gpu struct{
GpuName string `xml:"product_name"`
BiosVersion string `xml:"vbios_version"`
FanSpeed string `xml:"fan_speed"`
Utilization GpuUtilization `xml:"utilization"`
Temperature GpuTemperature `xml:"temperature"`
Clock GpuClock `xml:"clocks"`
}
type GpuUtilization struct {
GpuUsage string `xml:"gpu_util"`
MemoryUsage string `xml:"memory_util"`
}
type GpuTemperature struct {
GpuTemp string `xml:"gpu_temp"`
}
type GpuClock struct {
GpuClock string `xml:"graphics_clock"`
GpuMemClock string `xml:"mem_clock"`
}
#+end_src
** Docker Module
:PROPERTIES:
:CUSTOM_ID: docker-module
:END:
This section provides information on how the server module interacts
with the docker containers. The server calls 2 routes which either
creates or removes the docker container. Docker has a huge advantage
because it takes less than 20 seconds to spin up a new container once
it's built and executed at least once. For docker operations a separate
module/package has been created. The following subtopics will provide
more information on how this package works.
*** Docker Api
:PROPERTIES:
:CUSTOM_ID: docker-api
:END:
For this the api has been taken from the official docker repository. To
be more specific it is the client module in the official docker
repository. Docker was built using Go. During this project Docker
functions could be directly called from the docker repository. The
Docker api initially ensures that it can detect the docker environment
variables. Once detected, it can execute various functions from the
docker client module. The reason the docker api was selected was to
detect and handle errors better.
*** Docker Image
:PROPERTIES:
:CUSTOM_ID: docker-image
:END:
The docker image used to spin up the containers is called
ConSol/docker-headless-vnc-container. The following container was
modified to open SSH ports for an SSH connection. The following docker
image runs ubuntu 16. The reason this image was chosen as a default is
because if the client wants to access the container in the form of a
desktop environment. This image would allow the client to do so from
just a browser.
*** Build container
:PROPERTIES:
:CUSTOM_ID: build-container
:END:
This function pulls the docker image locally and builds the image.
Initially there is a timeout function to ensure that building the image
does not take too long to build. The next phase would be based on the
path to get the DockerFile. The tag name of the container is set as
p2p-ubuntu as default. Once the following is set then the docker build
command is executed.
*** Run container
:PROPERTIES:
:CUSTOM_ID: run-container
:END:
After building the container it needs to be executed for the user to
access the container and do certain operations. The docker
package/module has a function to do this. The function takes in the
docker environment as a parameter and also the docker struct. The docker
struct has information such as the TCP ports which are supposed to be
open and whether the docker container should have the GPU hooked to it
or not. Based on the appropriate information provided ,the docker image
gets started. The Image gets started by interacting with the docker
client modules. When hooking the GPU the docker run command is called
from the shell. This is because the docker Api does not support the GPU
module yet. When the container is executed for the first time it takes
more than 10 minutes to build. From the second time onwards it takes
only 10 seconds to run.
*** Stop and remove container
:PROPERTIES:
:CUSTOM_ID: stop-and-remove-container
:END:
This implementation here ensures that the docker is stopped, and the
container is removed. This is to ensure it does not utilize server
resources when it is not being used, or the task that is intended to be
executed is complete. To run this function all that is needed is the
docker container ID. If the function is successful it returns a string
that says success.
*** Ports json file
:PROPERTIES:
:CUSTOM_ID: ports-json-file
:END:
This file will help map internal ports inside a container to external
ports inside a container. A common example would be the SSH port which
is port 22 inside the docker container and is mapped to random TCP port
outside container so that any external machines can directly connect
into the container. The below representation mentions of where the
ports.json file is located and also the format of that file.
#+begin_example
|_ <Container name>
|_ Dockerfile
|_ description.txt
|_ ports.json // The ports file
#+end_example
Format of the ports.json file
#+begin_example
{
"Port": [
{
"PortName": "<Port name>",
"InternalPort": <internal port>,
"Type": "<tcp/udp>",
"ExternalPort": <external port>,
"IsUsed": "<boolean value (i.e true or false)>",
"Description": "<description about the port>"
}, ... n
]
}
#+end_example