diff --git a/Docs/index.html b/Docs/index.html index a91ef87..13fccc8 100644 --- a/Docs/index.html +++ b/Docs/index.html @@ -3,7 +3,7 @@ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
- +
-This project focuses on creating a framework for running heavy computational tasks that a regular -computer cannot handle easily. These tasks may include graphically demanding video games, rendering -3D animations, and performing complex protein folding simulations. The major focus of this project -is not on financial incentives but rather on building a robust and efficient peer-to-peer (P2P) -network to decentralise task execution and increase the computational bandwidth available for +This project focuses on creating a framework for running heavy computational tasks that a regular +computer cannot handle easily. These tasks may include graphically demanding video games, rendering +3D animations, and performing complex protein folding simulations. The major focus of this project +is not on financial incentives but rather on building a robust and efficient peer-to-peer (P2P) +network to decentralise task execution and increase the computational bandwidth available for such tasks.
-The P2PRC framework serves as a foundation for decentralised rendering and computation, -providing insights into how tasks can be distributed efficiently across a network of peers. -Leveraging the P2PRC approach, this project extends its capabilities to handle a +The P2PRC framework serves as a foundation for decentralised rendering and computation, +providing insights into how tasks can be distributed efficiently across a network of peers. +Leveraging the P2PRC approach, this project extends its capabilities to handle a wider range of computationally intensive tasks.
+// Do ensure that the docker command does not need sudo to run sudo chmod 666 /var/run/docker.sock@@ -279,7 +285,7 @@ To set up the internal dependencies and build the entire go code into a single binary -+make
+.\install.bat
.bashrc
+
export P2PRC=/<PATH>/p2p-rendering-computation
export PATH=/<PATH>/p2p-rendering-computation:${PATH}
@@ -310,14 +316,14 @@ export PATH=/<PATH>/p2p-rendering-computation:${PATH}
+Nix is a growing ecosystem that allows flexibility on how you develop, build and package software and configurations. It brings all programming languages (and all other tooling) to an equal footing, despite deep design differences. More importantly, integrates all the "packaging" into the context of a "pure" function. +
+ + ++P2PRC aims to become a utility that can be used in various flexible manners and having Nix support is a good alternative to accomplish this goal. +
+ + ++Nix Flake is a format, within the Nix ecosystem, intentionally designed to encourage a standard in packaging distribution. The current packaging assumes that you have "nix flake" installed because it is currently an experimental feature of Nix. +
++In case you want to develop, build or integrate using nix, you just need to run either "nix develop" or "nix run" from the command line locally in a cloned git repository or by running "nix run github:akilan1999/p2p-rendering-computation – –help" +
+ + ++P2PRC library also is ready to be imported into other nix flakes. To accomplish that please make sure to override the target nixpkgs environment in the following manner; +
+ +
+pkgs = import nixpkgs {
+ inherit system;
+ overlays = [
+ p2prc-flake.overlays.default
+ ];
+};
+
+
+
++This will make the p2prc executable available in the environment of any application you use. +
++The project is structured to provide language bindings to any programming language. The first one being supported in this manner is the Haskell programming language. It provides a bootstrapping script for a new Cabal project with p2prc binary available in the environment and, more relevantly, the Haskell library bindings available in the virtual environment cabal environment. +
+ ++nix run git+https://github.com/akilan1999/p2p-rendering-computation#initHaskellProject -- <PROJECT_NAME> ++ + +
+This will generate a new haskell project setup to automatically work with the p2prc development and running environment. +
+ + ++Once completed, you should go into the project directory and copy the nix flake template, necessary to define the project's environment. +
+ ++nix flake init -t github:akilan1999/p2p-rendering-computation#haskell ++ +
+The previous command sets up the flake environment and its dependencies. It will look like the following +
+ +
+{
+ description = "Start of Haskell P2PRC flake";
+
+ inputs =
+ {
+ nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+
+ flake-util.url = "github:numtide/flake-utils";
+
+ p2prc-flake.url = "github:akilan1999/p2p-rendering-computation";
+ };
+
+ outputs = { nixpkgs, p2prc-flake, flake-utils, ... }:
+ (flake-utils.lib.eachDefaultSystem (system:
+ let
+
+ pkgs = import nixpkgs {
+ inherit system;
+ overlays = [
+ p2prc-flake.overlays.default
+ p2prc-flake.overlays.bindings
+ ];
+ };
+
+ in {
+
+ packages.default = pkgs.haskellPackages.callPackage ./cabal.nix { };
+
+ devShells.default = pkgs.haskellPackages.shellFor {
+
+ packages = p: [
+ (p.callPackage ./cabal.nix { })
+ ];
+
+ buildInputs = with pkgs; [
+ p2prc-flake.packages.${system}.default
+ ghc
+ cabal2nix
+ cabal-install
+ ];
+
+ shellHook = ''
+ cabal2nix . > ./cabal.nix
+ '';
+ };
+ }
+ ));
+}
+
+
++The template uses Cabal2Nix which manages the Haskell virtual environment automatically. It applies the system overlays, sets up the shell environment for the project (updating the Cabal2Nix configuration) and packages the main executable. +
+-p2prc -s ++p2prc -s
+p2prc --specs=<ip address>
use the --gpu if you know the other machine has a gpu.
+p2prc --touch=<server ip address> -p <number of ports> --gpu
The docker id is present in the output where you create a container
--p2prc --rm=<server ip address> --id=<docker container id> ++p2prc --rm=<server ip address> --id=<docker container id>
-p2prc --as=<server ip address you want to add> ++p2prc --as=<server ip address you want to add>
-p2prc --us ++p2prc --us
-p2prc --ls ++p2prc --ls
+p2prc --ni
+p2prc --tc
+p2prc --plugin <plugin name> --id <container id or group id>
+p2prc --cgroup
+p2prc --group <group id> --id <container id>
+p2prc --groups
+p2prc --group <group id>
-p2prc --rmcgroup --group <group id> --id <container id> ++p2prc --rmcgroup --group <group id> --id <container id>
+p2prc --rmgroup <group id>
+p2prc --pp <repo link>
-p2prc --rp <plugin name> ++p2prc --rp <plugin name>
+p2prc --amd "custom metadata"
-Allows to expose remote ports from a machine in the P2P network. +Allows to expose remote ports from a machine in the P2P network.
-+p2prc --mp <port no to map> --dn <domain name to link Mapped port against>
-This is to ensure ports on remote machines on the P2PRC can be easily opened. +This is to ensure ports on remote machines on the P2PRC can be easily opened.
-+p2prc --mp <port no to map> --dn <domain name to link Mapped port against> --ra <remote server address>@@ -561,28 +697,28 @@ p2prc --mp <port no to map> --dn <domain name to link Mapped port again
-Adds a root node to P2RRC and overwrites all other nodes in the ip table. +Adds a root node to P2RRC and overwrites all other nodes in the ip table. To be only added before the network is started and with the intention of a fresh instance.
-+p2prc --arn --ip <root node ip address> -p <root node port no>
-The P2P module is for managing server information within the network. -It maintains and updates the IP table, ensuring accuracy by preventing duplicates and removing -entries for unreachable servers. Furthermore, the module conducts speed tests on the listed servers -to determine upload and download speeds. This valuable information enables users to identify nearby +The P2P module is for managing server information within the network. +It maintains and updates the IP table, ensuring accuracy by preventing duplicates and removing +entries for unreachable servers. Furthermore, the module conducts speed tests on the listed servers +to determine upload and download speeds. This valuable information enables users to identify nearby servers with optimal performance, enhancing their overall network experience.
@@ -603,7 +739,7 @@ implementations of each functionality in depth.The ip table file is a json as the format with a list of servers ip @@ -631,7 +767,7 @@ configuration module. "NAT": "<boolean representing if the node is behind NAT or not>", "EscapeImplementation": "<NAT traversal implementation>", "ProxyServer": "<If the node listed is acting as a proxy server>", - "UnSafeMode": <Unsafe mode if turned on will allow all nodes in the network public keys to be + "UnSafeMode": <Unsafe mode if turned on will allow all nodes in the network public keys to be added to that particular node>", "PublicKey": "<Public key of that particular node>", "CustomInformation": "<custom information passed in through all the nodes in the network>" @@ -642,7 +778,7 @@ configuration module.
The latency is measured in milliseconds. The route /serverinfo is @@ -653,7 +789,7 @@ recorded.
P2PRC currently supports TURN for NAT traversal. @@ -661,7 +797,7 @@ P2PRC currently supports TURN for NAT traversal.
The current TURN implementation used is FRP. The TURN server is also @@ -674,13 +810,13 @@ from the TURN server. The flow below describes the workflow.
/FRPPort+http://<turn server ip>:<server port no>/FRPport@@ -714,7 +850,7 @@ func main() {
Language bindings @@ -725,7 +861,7 @@ compiler.
Below are a sample set of commands to open the bindings implementation.
-+# run cd Bindings/ # list files -ls +ls # search for file Client.go
There a few things to notice which are different from your standard Go @@ -857,7 +993,7 @@ that convert a go object to JSON string initially and converts it to
+
func <Function name> (output *C.char) {
<response>,<error> := <P2PRC function name>(<parameters if needed>)
if <error> != nil {
@@ -872,12 +1008,12 @@ func <Function name> (output *C.char) {
P2PRC officially supports Haskell bindings and will further support project using Haskell to build orchestrators on top of P2PRC. @@ -929,7 +1065,7 @@ project using Haskell to build orchestrators on top of P2PRC.
The configuration module is responsible to store basic information of @@ -978,14 +1114,14 @@ configuration file was generated using JSON format.
The Abstractions package consists of black-boxed functions for P2PRC.
Init(<Project name>): Initializes P2PRC with all the needed
@@ -1008,14 +1144,14 @@ faster.P2PRC currently supports TURN for NAT traversal.
The current TURN implementation used is FRP. The TURN server is also @@ -1030,13 +1166,13 @@ from the TURN server. The flow below describes the workflow.
/FRPPort+http://<turn server ip>:<server port no>/FRPport@@ -1067,9 +1203,9 @@ func main() {
The simulation library is a really simple library to locally run serveral instances of P2PRC to test the functionality @@ -1078,9 +1214,9 @@ the Node discovery mechanism. There are current 5 commands which can be used as shown below:
# Unset default P2PRC env path unset P2PRC @@ -1126,13 +1262,13 @@ Source code: -12. Blog posts
-++14. Blog posts
+--12.1. Self host within 5 minutes any program
-++14.1. Self host within 5 minutes any program
+-
- Author: Akilan Selvacoumar
- Date: 28-01-2025
@@ -1143,7 +1279,7 @@ Source code: + @@ -1153,9 +1289,9 @@ This is a fun expirement for anyone to try to quickly run a server and quickly do a map port and domain name mapping in a single command.-12.1.1. 1. Find a program you want to run
-++-14.1.1. 1. Find a program you want to run
+Let's try to setup a really easy program (Let's do with Linkwarden with docker compose :) ). This is under the assumption you have docker @@ -1163,8 +1299,8 @@ compose installed on your local machine.
-
- Let's run Linkwarden using docker compose and P2PRC
-+- Let's run Linkwarden using docker compose and P2PRC
+@@ -1252,17 +1388,17 @@ ex: https://linkwarden.akilan.io-13. Ideas for future potencial features
-++15. Ideas for future potencial features
+-Consists of personal loideas for the future of P2PRC. At moment only has main contributors writiing to this.
-13.1. To support hetrogenous set of Nodes that cannot run P2PRC
-++15.1. To support hetrogenous set of Nodes that cannot run P2PRC
+-This stems from a personal issue I have when doing research on CheriBSD kernel. For my research I am using the ARM morello @@ -1277,9 +1413,9 @@ and would introduce a new layer fault tolerance within a local network nodes.
-13.1.1. Assumptions:
-++-15.1.1. Assumptions:
+
- I have a Morello board that cannot run P2PRC
- The Morello has a local IP address (ex: 192.168.0.10)
@@ -1291,9 +1427,9 @@ as well look into custom tasks into the hetrogenous hardware.-13.1.2. Set of interesting possible:
-++-15.1.2. Set of interesting possible:
+We build a cool set possibilities before and use this to build up the implementation plan. @@ -1309,9 +1445,9 @@ weight hypervisors.
-13.1.3. Implementation
-++15.1.3. Implementation
+-
- To use implementations similar to socat to ensure we can bind address of local nodes to a node running P2PRC and the node running P2PRC can do a local map port.
@@ -1324,7 +1460,7 @@ other nodes can access the Morello board who have permission access.+diff --git a/Docs/index.org b/Docs/index.org index 98e79c5..ac44b3a 100644 --- a/Docs/index.org +++ b/Docs/index.org @@ -17,16 +17,16 @@ :PROPERTIES: :CUSTOM_ID: chapter1-introduction :END: -This project focuses on creating a framework for running heavy computational tasks that a regular -computer cannot handle easily. These tasks may include graphically demanding video games, rendering -3D animations, and performing complex protein folding simulations. The major focus of this project -is not on financial incentives but rather on building a robust and efficient peer-to-peer (P2P) -network to decentralise task execution and increase the computational bandwidth available for +This project focuses on creating a framework for running heavy computational tasks that a regular +computer cannot handle easily. These tasks may include graphically demanding video games, rendering +3D animations, and performing complex protein folding simulations. The major focus of this project +is not on financial incentives but rather on building a robust and efficient peer-to-peer (P2P) +network to decentralise task execution and increase the computational bandwidth available for such tasks. -The P2PRC framework serves as a foundation for decentralised rendering and computation, -providing insights into how tasks can be distributed efficiently across a network of peers. -Leveraging the P2PRC approach, this project extends its capabilities to handle a +The P2PRC framework serves as a foundation for decentralised rendering and computation, +providing insights into how tasks can be distributed efficiently across a network of peers. +Leveraging the P2PRC approach, this project extends its capabilities to handle a wider range of computationally intensive tasks. ** Motivation @@ -182,6 +182,110 @@ GLOBAL OPTIONS: -------------- +* Nix Flake + +Nix is a growing ecosystem that allows flexibility on how you develop, build and package software and configurations. It brings all programming languages (and all other tooling) to an equal footing, despite deep design differences. More importantly, integrates all the "packaging" into the context of a "pure" function. + + +P2PRC aims to become a utility that can be used in various flexible manners and having Nix support is a good alternative to accomplish this goal. + + +Nix Flake is a format, within the Nix ecosystem, intentionally designed to encourage a standard in packaging distribution. The current packaging assumes that you have "nix flake" installed because it is currently an experimental feature of Nix. + + +** P2PRC core Go language repo + +In case you want to develop, build or integrate using nix, you just need to run either "nix develop" or "nix run" from the command line locally in a cloned git repository or by running "nix run github:akilan1999/p2p-rendering-computation -- --help" + + +P2PRC library also is ready to be imported into other nix flakes. To accomplish that please make sure to override the target nixpkgs environment in the following manner; + +#+begin_example +pkgs = import nixpkgs { + inherit system; + overlays = [ + p2prc-flake.overlays.default + ]; +}; +#+end_example + + +This will make the p2prc executable available in the environment of any application you use. + +* P2PRC Haskell library + + +The project is structured to provide language bindings to any programming language. The first one being supported in this manner is the Haskell programming language. It provides a bootstrapping script for a new Cabal project with p2prc binary available in the environment and, more relevantly, the Haskell library bindings available in the virtual environment cabal environment. + +#+begin_example +nix run git+https://github.com/akilan1999/p2p-rendering-computation#initHaskellProject --
![]()
Figure 1: Implementation idea (To be improved upon)
@@ -1336,7 +1472,7 @@ other nodes can access the Morello board who have permission access.+#+end_example + + +This will generate a new haskell project setup to automatically work with the p2prc development and running environment. + + +Once completed, you should go into the project directory and copy the nix flake template, necessary to define the project's environment. + +#+begin_example +nix flake init -t github:akilan1999/p2p-rendering-computation#haskell +#+end_example + +The previous command sets up the flake environment and its dependencies. It will look like the following + +#+begin_example +{ + description = "Start of Haskell P2PRC flake"; + + inputs = + { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + + flake-util.url = "github:numtide/flake-utils"; + + p2prc-flake.url = "github:akilan1999/p2p-rendering-computation"; + }; + + outputs = { nixpkgs, p2prc-flake, flake-utils, ... }: + (flake-utils.lib.eachDefaultSystem (system: + let + + pkgs = import nixpkgs { + inherit system; + overlays = [ + p2prc-flake.overlays.default + p2prc-flake.overlays.bindings + ]; + }; + + in { + + packages.default = pkgs.haskellPackages.callPackage ./cabal.nix { }; + + devShells.default = pkgs.haskellPackages.shellFor { + + packages = p: [ + (p.callPackage ./cabal.nix { }) + ]; + + buildInputs = with pkgs; [ + p2prc-flake.packages.${system}.default + ghc + cabal2nix + cabal-install + ]; + + shellHook = '' + cabal2nix . > ./cabal.nix + ''; + }; + } + )); +} +#+end_example + +The template uses Cabal2Nix which manages the Haskell virtual environment automatically. It applies the system overlays, sets up the shell environment for the project (updating the Cabal2Nix configuration) and packages the main executable. + * Using basic commands :PROPERTIES: :CUSTOM_ID: using-basic-commands @@ -192,7 +296,7 @@ GLOBAL OPTIONS: :END: #+begin_example -p2prc -s +p2prc -s #+end_example *** View server Specification @@ -220,7 +324,7 @@ p2prc --touch= -p --gpu The docker id is present in the output where you create a container #+begin_example -p2prc --rm= --id= +p2prc --rm= --id= #+end_example *** Adding servers to ip table @@ -228,7 +332,7 @@ p2prc --rm= --id= :CUSTOM_ID: adding-servers-to-ip-table :END: #+begin_example -p2prc --as= +p2prc --as= #+end_example *** Update ip table @@ -236,7 +340,7 @@ p2prc --as= :CUSTOM_ID: update-ip-table :END: #+begin_example -p2prc --us +p2prc --us #+end_example *** List Servers @@ -244,7 +348,7 @@ p2prc --us :CUSTOM_ID: list-servers :END: #+begin_example -p2prc --ls +p2prc --ls #+end_example *** View Network interfaces @@ -308,7 +412,7 @@ p2prc --group :CUSTOM_ID: delete-container-from-group :END: #+begin_example -p2prc --rmcgroup --group --id +p2prc --rmcgroup --group --id #+end_example *** Delete entire group @@ -332,7 +436,7 @@ p2prc --pp :CUSTOM_ID: deleting-plugin-from-the-plugin-directory :END: #+begin_example -p2prc --rp +p2prc --rp #+end_example *** Added custom metadata about the current node @@ -344,12 +448,12 @@ p2prc --amd "custom metadata" #+end_example *** MapPort and link to domain name -Allows to expose remote ports from a machine in the P2P network. +Allows to expose remote ports from a machine in the P2P network. #+begin_example p2prc --mp --dn #+end_example **** MapPort in remote machine -This is to ensure ports on remote machines on the P2PRC can be easily opened. +This is to ensure ports on remote machines on the P2PRC can be easily opened. #+begin_example p2prc --mp --dn --ra #+end_example @@ -357,7 +461,7 @@ p2prc --mp --dn --ra -------------- *** Add root node -Adds a root node to P2RRC and overwrites all other nodes in the ip table. +Adds a root node to P2RRC and overwrites all other nodes in the ip table. To be only added before the network is started and with the intention of a fresh instance. #+begin_example @@ -368,10 +472,10 @@ p2prc --arn --ip -p :PROPERTIES: :CUSTOM_ID: p2p-module-implementation :END: -The P2P module is for managing server information within the network. -It maintains and updates the IP table, ensuring accuracy by preventing duplicates and removing -entries for unreachable servers. Furthermore, the module conducts speed tests on the listed servers -to determine upload and download speeds. This valuable information enables users to identify nearby +The P2P module is for managing server information within the network. +It maintains and updates the IP table, ensuring accuracy by preventing duplicates and removing +entries for unreachable servers. Furthermore, the module conducts speed tests on the listed servers +to determine upload and download speeds. This valuable information enables users to identify nearby servers with optimal performance, enhancing their overall network experience. #+caption: UML diagram of P2P module @@ -415,7 +519,7 @@ configuration module. "NAT": " ", "EscapeImplementation": " ", "ProxyServer": " ", - "UnSafeMode": ", "PublicKey": " ", "CustomInformation": " " @@ -538,7 +642,7 @@ Below are a sample set of commands to open the bindings implementation. # run cd Bindings/ # list files -ls +ls # search for file Client.go #+end_example @@ -653,7 +757,7 @@ cd Bindings/python/export/ # list files ls # Expected output -SharedObjects/ library.py requirements.txt +SharedObjects/ library.py requirements.txt #+end_src Above shows a generated folder which consists of a folder called @@ -742,7 +846,7 @@ The Abstractions package consists of black-boxed functions for P2PRC. - =ViewIPTable()=: View the IP table which about nodes in the network. - =UpdateIPTable()=: Force update IP table to learn about new nodes faster. - + * NAT Traversal :PROPERTIES: :CUSTOM_ID: nat-traversal @@ -803,7 +907,7 @@ which can be used as shown below: ** Command - ***P2PRC instances "number of instances"*** : Starts p2prc processes based on the number of number instances provided. This includes - creating temporary folders with independent IPTables. + creating temporary folders with independent IPTables. - ***Start all instances "number of instances"***: Starts the P2PRC instances created. This function enters into the instance directory and runs ```p2prc -s &``` with a delay for approximately 3 seconds. This allows @@ -811,8 +915,8 @@ which can be used as shown below: - ***IP Tables after Started number of instances"***: Prints the IPTable of each independant node. - ***Remove all test files***: Removes all test files created which needed for - testing the generated P2PRC instances. -- ***Kill all instances***: Kills all the background P2PRC instances created. + testing the generated P2PRC instances. +- ***Kill all instances***: Kills all the background P2PRC instances created. ** Sample bash instructions #+begin_src bash @@ -841,7 +945,7 @@ Source code: https://github.com/Akilan1999/p2p-rendering-computation/blob/master - Author: [[http://akilan.io/][Akilan Selvacoumar]] - Date: 28-01-2025 - + - Video tutorial: #+attr_html: :class video [[https://youtu.be/rN4SiVowg5E][https://i3.ytimg.com/vi/rN4SiVowg5E/maxresdefault.jpg]] @@ -853,7 +957,7 @@ quickly do a map port and domain name mapping in a single command. Let's try to setup a really easy program (Let's do with Linkwarden with docker compose :) ). This is under the assumption you have docker compose installed on your local machine. - + **** Let's run Linkwarden using docker compose and P2PRC [[https://docs.linkwarden.app/self-hosting/installation][Installation instructions]]: #+BEGIN_SRC @@ -909,14 +1013,14 @@ A entry 217.76.63.222 Your done now just head to the DOMAIN NAME you added. ex: https://linkwarden.akilan.io - - - - - - - + + + + + + + * Ideas for future potencial features @@ -953,7 +1057,7 @@ network nodes. - Spin servers on node not running P2PRC using the P2PRC standard abstractions. - Auto-setup P2PRC nodes with just SSH access via potencially a DSL. - A neat use case for CHERI for instance would be use the architecture to run light - weight hypervisors. + weight hypervisors. *** Implementation - To use implementations similar to [[https://linux.die.net/man/1/socat][socat]] to ensure we can bind address of local nodes to a node running P2PRC and the node running P2PRC can do a local map port.