did basic conversion to org mode for the docs folder
This commit is contained in:
278
Docs/ServerImplementation.md.org
Normal file
278
Docs/ServerImplementation.md.org
Normal file
@@ -0,0 +1,278 @@
|
||||
* Server Module Implementation
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: server-module-implementation
|
||||
:END:
|
||||
This section focuses on an in-depth understanding of the server module
|
||||
implementation. To understand the architecture of the server module
|
||||
refer. The server module can be split into various sections. Each
|
||||
section will provide information on how a certain feature works.
|
||||
|
||||
The server module takes care of setting and removing the virtualization
|
||||
environment (i.e containers) for accessing and doing the appropriate
|
||||
computation. It also interacts with the peer to peer module to update
|
||||
the IP table on the server side. The server module accesses information
|
||||
regarding CPU and GPU specifications of the machine running the server
|
||||
module. To do Speed tests the server has routes which allows it to
|
||||
upload and download a 50mb.
|
||||
|
||||
#+caption: UML diagram of server module
|
||||
[[file:images/servermoduleArch.png]]
|
||||
|
||||
** Web framework
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: web-framework
|
||||
:END:
|
||||
The web framework used for the server module is called Gin. The reason
|
||||
Gin was chosen is due to its wide use and strong documentation available
|
||||
on the official github repository. The default port used is 8088. For
|
||||
version 1.0 of the project ,the server needs to keep port 8088 open to
|
||||
ensure that other clients and servers can detect it. The possible
|
||||
requests available are GET and POST for this implementation. The
|
||||
possible responses are either a string or json response or a file. In
|
||||
the majority of routes a string response refers to an error when calling
|
||||
the following routes. The following sub topics below will talk about the
|
||||
route implemented:
|
||||
|
||||
*** /server_info
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: server_info
|
||||
:END:
|
||||
This route is responsible to get information about the specifications of
|
||||
the server. The response of this route is in json if the call was
|
||||
successful.
|
||||
|
||||
*** /50
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: section
|
||||
:END:
|
||||
This route is responsible for returning a randomly generated 50mb file.
|
||||
This is used to calculate the download speed from the p2p module.
|
||||
|
||||
*** /IpTable
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: iptable
|
||||
:END:
|
||||
This route is a POST request that is responsible to update the server IP
|
||||
table based on the IP table the client provides. Once the server gets
|
||||
the IP table it checks if the client is also a server. This is done by
|
||||
calling the url http://:8088/server_info. If the server_info route from
|
||||
the client responds back with computer specifications of the client.
|
||||
Then the server initially appends the clients IP to the struct. After
|
||||
that the IP table received from the client is uploaded to the struct.
|
||||
Once this is done the server passes the struct to the peer to peer
|
||||
module function. The peer to peer module function will return the back
|
||||
with the new struct with the valid server nodes. The server responds
|
||||
back to the new struct as a json format. If a string is present in the
|
||||
response then there is probably an error on the server side.
|
||||
|
||||
*** /startcontainer
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: startcontainer
|
||||
:END:
|
||||
This route takes in a GET request with the number of TCP ports to open
|
||||
and checks whether the docker container should be hooked to the GPU or
|
||||
not. This route talks to the docker module implemented as a sub module
|
||||
in the server module. More information on the docker module in section
|
||||
5.4.3. This route calls docker the module to start the container for the
|
||||
client. The docker module returns back a struct. This struct is returned
|
||||
back to the client as the json response. This struct consists of
|
||||
information such as docker id, ports numbers open , information
|
||||
regarding SSH and VNC connections to the docker container created when
|
||||
the client created this request.
|
||||
|
||||
*** /RemoveContainer
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: removecontainer
|
||||
:END:
|
||||
This route takes in a GET request as the container ID. Based on the
|
||||
container ID provided ,it calls the docker module which deletes the
|
||||
container. If the deletion is successful it returns back a string which
|
||||
says success.
|
||||
|
||||
** Server information/ Specification
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: server-information-specification
|
||||
:END:
|
||||
This section provides information on how the server specifications are
|
||||
read. There are 2 major implementations. The first implementation
|
||||
mentions how basic information such as RAM usage, CPU specification are
|
||||
detected and the second implementation mentions how the GPU drivers are
|
||||
detected and information is extracted. The client has to assume that the
|
||||
server is using default docker settings in terms of CPU cycles and other
|
||||
parameters.
|
||||
|
||||
*** Basic Information
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: basic-information
|
||||
:END:
|
||||
The file name for these functions is called gopsutil.go. This codebase
|
||||
uses the library gopsutil. Gopsutil has various packages or modules
|
||||
within the library which have functions implemented to get system
|
||||
information. The following information is stored in a struct and the
|
||||
function returns that struct.
|
||||
|
||||
#+begin_src go
|
||||
type SysInfo struct {
|
||||
Hostname string `bson:hostname`
|
||||
Platform string `bson:platform`
|
||||
CPU string `bson:cpu`
|
||||
RAM uint64 `bson:ram`
|
||||
Disk uint64 `bson:disk`
|
||||
GPU *Query `xml: GpuInfo`
|
||||
}
|
||||
#+end_src
|
||||
|
||||
*** GPU Information
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: gpu-information
|
||||
:END:
|
||||
The file name for these functions is called GPU.go. This codebase checks
|
||||
if the Nvidia driver exists and returns the driver information. To do
|
||||
this a shell command called nvidia-smi is executed. This shell command
|
||||
is executed with a --xml as flag to ensure that the output is in the XML
|
||||
format. If there is an output as a xml format, that means there is an
|
||||
nvidia driver installed, and the function just reads the output and
|
||||
stores it to the struct and returns the GPU information.
|
||||
|
||||
#+begin_src go
|
||||
|
||||
type Query struct {
|
||||
DriveVersion string `xml:"driver_version"`
|
||||
Gpu Gpu `xml:"gpu"`
|
||||
}
|
||||
|
||||
type Gpu struct{
|
||||
GpuName string `xml:"product_name"`
|
||||
BiosVersion string `xml:"vbios_version"`
|
||||
FanSpeed string `xml:"fan_speed"`
|
||||
Utilization GpuUtilization `xml:"utilization"`
|
||||
Temperature GpuTemperature `xml:"temperature"`
|
||||
Clock GpuClock `xml:"clocks"`
|
||||
}
|
||||
|
||||
type GpuUtilization struct {
|
||||
GpuUsage string `xml:"gpu_util"`
|
||||
MemoryUsage string `xml:"memory_util"`
|
||||
}
|
||||
|
||||
type GpuTemperature struct {
|
||||
GpuTemp string `xml:"gpu_temp"`
|
||||
}
|
||||
|
||||
type GpuClock struct {
|
||||
GpuClock string `xml:"graphics_clock"`
|
||||
GpuMemClock string `xml:"mem_clock"`
|
||||
}
|
||||
#+end_src
|
||||
|
||||
** Docker Module
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: docker-module
|
||||
:END:
|
||||
This section provides information on how the server module interacts
|
||||
with the docker containers. The server calls 2 routes which either
|
||||
creates or removes the docker container. Docker has a huge advantage
|
||||
because it takes less than 20 seconds to spin up a new container once
|
||||
it's built and executed at least once. For docker operations a separate
|
||||
module/package has been created. The following subtopics will provide
|
||||
more information on how this package works.
|
||||
|
||||
*** Docker Api
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: docker-api
|
||||
:END:
|
||||
For this the api has been taken from the official docker repository. To
|
||||
be more specific it is the client module in the official docker
|
||||
repository. Docker was built using Go. During this project Docker
|
||||
functions could be directly called from the docker repository. The
|
||||
Docker api initially ensures that it can detect the docker environment
|
||||
variables. Once detected, it can execute various functions from the
|
||||
docker client module. The reason the docker api was selected was to
|
||||
detect and handle errors better.
|
||||
|
||||
*** Docker Image
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: docker-image
|
||||
:END:
|
||||
The docker image used to spin up the containers is called
|
||||
ConSol/docker-headless-vnc-container. The following container was
|
||||
modified to open SSH ports for an SSH connection. The following docker
|
||||
image runs ubuntu 16. The reason this image was chosen as a default is
|
||||
because if the client wants to access the container in the form of a
|
||||
desktop environment. This image would allow the client to do so from
|
||||
just a browser.
|
||||
|
||||
*** Build container
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: build-container
|
||||
:END:
|
||||
This function pulls the docker image locally and builds the image.
|
||||
Initially there is a timeout function to ensure that building the image
|
||||
does not take too long to build. The next phase would be based on the
|
||||
path to get the DockerFile. The tag name of the container is set as
|
||||
p2p-ubuntu as default. Once the following is set then the docker build
|
||||
command is executed.
|
||||
|
||||
*** Run container
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: run-container
|
||||
:END:
|
||||
After building the container it needs to be executed for the user to
|
||||
access the container and do certain operations. The docker
|
||||
package/module has a function to do this. The function takes in the
|
||||
docker environment as a parameter and also the docker struct. The docker
|
||||
struct has information such as the TCP ports which are supposed to be
|
||||
open and whether the docker container should have the GPU hooked to it
|
||||
or not. Based on the appropriate information provided ,the docker image
|
||||
gets started. The Image gets started by interacting with the docker
|
||||
client modules. When hooking the GPU the docker run command is called
|
||||
from the shell. This is because the docker Api does not support the GPU
|
||||
module yet. When the container is executed for the first time it takes
|
||||
more than 10 minutes to build. From the second time onwards it takes
|
||||
only 10 seconds to run.
|
||||
|
||||
*** Stop and remove container
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: stop-and-remove-container
|
||||
:END:
|
||||
This implementation here ensures that the docker is stopped, and the
|
||||
container is removed. This is to ensure it does not utilize server
|
||||
resources when it is not being used, or the task that is intended to be
|
||||
executed is complete. To run this function all that is needed is the
|
||||
docker container ID. If the function is successful it returns a string
|
||||
that says success.
|
||||
|
||||
*** Ports json file
|
||||
:PROPERTIES:
|
||||
:CUSTOM_ID: ports-json-file
|
||||
:END:
|
||||
This file will help map internal ports inside a container to external
|
||||
ports inside a container. A common example would be the SSH port which
|
||||
is port 22 inside the docker container and is mapped to random TCP port
|
||||
outside container so that any external machines can directly connect
|
||||
into the container. The below representation mentions of where the
|
||||
ports.json file is located and also the format of that file.
|
||||
|
||||
#+begin_example
|
||||
|_ <Container name>
|
||||
|_ Dockerfile
|
||||
|_ description.txt
|
||||
|_ ports.json // The ports file
|
||||
#+end_example
|
||||
|
||||
Format of the ports.json file
|
||||
|
||||
#+begin_example
|
||||
{
|
||||
"Port": [
|
||||
{
|
||||
"PortName": "<Port name>",
|
||||
"InternalPort": <internal port>,
|
||||
"Type": "<tcp/udp>",
|
||||
"ExternalPort": <external port>,
|
||||
"IsUsed": "<boolean value (i.e true or false)>",
|
||||
"Description": "<description about the port>"
|
||||
}, ... n
|
||||
]
|
||||
}
|
||||
#+end_example
|
||||
Reference in New Issue
Block a user