Merge pull request #69 from Akilan1999/customize-p2prc
Extension p2prc you create your own modification
This commit is contained in:
Akilan Selvacoumar
5
.gitignore
vendored
5
.gitignore
vendored
@@ -14,4 +14,7 @@ p2p/iptable/
|
||||
#ignore plugins added
|
||||
plugin/deploy/
|
||||
#ignore track container file
|
||||
client/trackcontainers/
|
||||
client/trackcontainers/
|
||||
# Test generated files
|
||||
generate/p2prctest
|
||||
generate/Test
|
||||
116
Docs/GenerateImplementation.md
Normal file
116
Docs/GenerateImplementation.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# Generate Module
|
||||
P2PRC is a great layer of abstraction. This means that in many cases it is not an end product but rather
|
||||
a tool that customized as an end product. An example would be writing your own billing module to monetize
|
||||
the computation power available. The generate module copies the current with the appropriate git histories
|
||||
and keeps only the go files which would be useful to edit. To use the generate module the user will need
|
||||
to have a go compiler present in his computer. Due to the introduction of this module there will 2 releases:
|
||||
|
||||
- Regular Release (Consists of only the build binary and cli command cannot access the generate module)
|
||||
- Developer Release (Consists of important Go files and the cli can access the generate module)
|
||||
|
||||
## How does this work ?
|
||||
|
||||
### [Struct information](https://github.com/Akilan1999/p2p-rendering-computation/blob/9d69aed8ce0fe5273aaff2828f7d51c3d5ac2ce4/generate/generate.go#L19)
|
||||
- ###```Generate.go```:
|
||||
This file creates a local copy of P2PRC from where the CLI was called from.
|
||||
This go file also does various stuff like instruction of file should be ignored when copying and
|
||||
which of should not be. Now let's understand this. Below is a sample code which does the following:
|
||||
|
||||
```go
|
||||
//----------------------------------------------------------------
|
||||
// Action performed:
|
||||
// - Ensuring main.go file exists
|
||||
// - Skipping all .go files apart from the ones listed above
|
||||
// - Skipping .idea/ directory
|
||||
// - Skipping Makefile file
|
||||
//----------------------------------------------------------------
|
||||
Options.Skip = func(src string) (bool, error) {
|
||||
switch {
|
||||
case strings.HasSuffix(src, "main.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, ".go"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, ".idea"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, "Makefile"):
|
||||
return true, nil
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Doing the copy
|
||||
err = copy.Copy("<P2PRC folder you want to copy from>", "<PATH to the directory>", Options)
|
||||
```
|
||||
|
||||
Unfortunately currently this will have to be manually edited in the ```Generate.go``` file. When using the generate
|
||||
module the user also creates their own Go module which is the modified version of P2PRC. This means
|
||||
if the 1 modified package is using another modified package then the appropriate import have to be modified
|
||||
in the file where the import is called:
|
||||
|
||||
Ex:
|
||||
```go
|
||||
//Sample Project module name = Test
|
||||
//Package names:
|
||||
//- Test/Genius
|
||||
//- Test/GeGeGenuis
|
||||
//
|
||||
// When we call the generate function with the new project with the module name = MicDrop
|
||||
// The new package name would be:
|
||||
// - MicDrop/Genius
|
||||
// - MicDrop/GeGeGenuis
|
||||
|
||||
// Test/Genius code depends on the package Test/GeGeGenuis
|
||||
import (
|
||||
"Test/GeGeGenuis"
|
||||
)
|
||||
|
||||
// When we create a new module with the copy of the
|
||||
// existing project we need change:
|
||||
import (
|
||||
"MicDrop/GeGeGenuis"
|
||||
)
|
||||
```
|
||||
|
||||
To do this we have built functions which can modify import names in the Go file provided.
|
||||
To customize the use case of your generate module you would need to manually add your own
|
||||
imports which are supposed to be replaced and in which files they are supposed to be replaced
|
||||
in.
|
||||
|
||||
```go
|
||||
// 1.0 - Test/Genius.go -> GeGeGenuis module
|
||||
// a is struct of type NewProject
|
||||
a.FileNameAST = "<path to project to copy from>/Test/Genius.go"
|
||||
// Get AST information of the file
|
||||
err := a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports("Test/GeGeGenuis", "MicDrop/GeGeGenuis")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
```
|
||||
|
||||
Higher order of execution of ```Generate.go```:
|
||||
1. Copy entire P2PRC project and ignores files which are not meant to be copied
|
||||
2. The folder name will be based on the new project name and the module name based on the new
|
||||
module name provided.
|
||||
3. Modifies the appropriate imports in the project as instructed in the code.
|
||||
4. Creates a commit with the new changes in the new project.
|
||||
|
||||
|
||||
- ### ``` modifyGenerate.go```:
|
||||
This a really simple implementation where we replace the imports
|
||||
in certain files as instructed from ```generate.go```. To do we create an AST (i.e Abstract Syntax tree)
|
||||
from new file we want to change the imports in. AST create a tree structure of expression. To change the
|
||||
import we can just traverse to the appropriate expression and change the value of that expression in
|
||||
the case of modifying imports. This approach is more simple than using templates.
|
||||
|
||||
|
||||
@@ -16,4 +16,5 @@ the project is git.sr.ht/~akilan1999/p2p-rendering-computation.
|
||||
## [Server Module](ServerImplementation.md)
|
||||
## [Client Module](ClientImplementation.md)
|
||||
## [P2P Module](P2PImplementation.md)
|
||||
## [Plugin Module](PluginImplementation.md)
|
||||
## [Plugin Module](PluginImplementation.md)
|
||||
## [Generate Module](GenerateImplementation.md)
|
||||
@@ -171,8 +171,14 @@ p2prc --rmcgroup --group <group id> --id <container id>
|
||||
```
|
||||
p2prc --rmgroup <group id>
|
||||
```
|
||||
|
||||
[read more on grouping containers](ClientImplementation.md#Grouping-Containers)
|
||||
### Extending usecase of P2PRC (Requires a go compiler to run)
|
||||
```
|
||||
p2prc --gen <project name> --mod <go module name>
|
||||
```
|
||||
[read more about the generate module](GenerateImplementation.md)
|
||||
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
@@ -13,5 +13,6 @@
|
||||
4. [Config Module](ConfigImplementation.md)
|
||||
5. [Cli Module](CliImplementation.md)
|
||||
6. [Plugin Module](PluginImplementation.md)
|
||||
5. [Problems](Problems.md)
|
||||
7. [Generate Module](GenerateImplementation.md)
|
||||
5. [Problems](https://github.com/Akilan1999/p2p-rendering-computation/issues)
|
||||
|
||||
|
||||
11
Makefile
11
Makefile
@@ -1,16 +1,7 @@
|
||||
SHELL := /bin/bash
|
||||
|
||||
install:
|
||||
go build -o p2prc
|
||||
echo '# Paths for p2p rendering and computation'
|
||||
echo 'export P2PRC=${PWD}'
|
||||
echo 'export PATH=${PWD}:$${PATH}'
|
||||
|
||||
build:
|
||||
go build -o p2prc
|
||||
|
||||
configfile:
|
||||
./p2prc --SetDefaultConfig
|
||||
sh install.sh p2prc
|
||||
|
||||
testcases:
|
||||
sh plugin/generate_test_case.sh
|
||||
|
||||
@@ -65,11 +65,7 @@ This project aims to create a peer to peer (p2p) network, where a user can use t
|
||||
// Add them to your .bashrc file
|
||||
export P2PRC=/<path>/p2p-rendering-computation
|
||||
export PATH=/<path>/p2p-rendering-computation:${PATH}
|
||||
|
||||
5. Generate Config file
|
||||
```
|
||||
make configfile
|
||||
```
|
||||
|
||||
|
||||
5. Test if it works
|
||||
```
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/client"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/config"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/generate"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/p2p"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/plugin"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/server"
|
||||
@@ -243,6 +244,32 @@ var CliAction = func(ctx *cli.Context) error {
|
||||
client.PrettyPrint(groups)
|
||||
}
|
||||
}
|
||||
// -- REMOVE ON REGULAR RELEASE --
|
||||
// when flag --gen is called an extension
|
||||
// of the project is created to repurpose
|
||||
// the project for custom purpose
|
||||
if Generate != "" {
|
||||
var err error
|
||||
// If the module name is provided
|
||||
if Modulename != "" {
|
||||
err = generate.GenerateNewProject(Generate,Modulename)
|
||||
} else {
|
||||
err = generate.GenerateNewProject(Generate,Generate)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
} else {
|
||||
fmt.Println("Created new folder: " + Generate)
|
||||
fmt.Println("1. Enter inside " + Generate + " directory")
|
||||
fmt.Println("2. git remote add " + Generate + " <PATH to the github repo>")
|
||||
fmt.Println("3. git push " + Generate + " <PATH to the github repo>")
|
||||
fmt.Println("4. go mod tidy")
|
||||
fmt.Println("5. sh install.sh " + Generate)
|
||||
fmt.Println("6. ./" + Generate + " -h (This is to test if the binary is working)")
|
||||
}
|
||||
}
|
||||
//--------------------------------
|
||||
|
||||
|
||||
|
||||
return nil
|
||||
|
||||
22
cmd/flags.go
22
cmd/flags.go
@@ -28,6 +28,11 @@ var (
|
||||
Groups bool
|
||||
RemoveContainerGroup bool
|
||||
RemoveGroup string
|
||||
// Generate only allowed in dev release
|
||||
// -- REMOVE ON REGULAR RELEASE --
|
||||
Generate string
|
||||
Modulename string
|
||||
//--------------------------------
|
||||
)
|
||||
|
||||
var AppConfigFlags = []cli.Flag{
|
||||
@@ -187,4 +192,21 @@ var AppConfigFlags = []cli.Flag{
|
||||
EnvVars: []string{"REMOVE_GROUP"},
|
||||
Destination: &RemoveGroup,
|
||||
},
|
||||
// Generate only allowed in dev release
|
||||
// -- REMOVE ON REGULAR RELEASE --
|
||||
&cli.StringFlag{
|
||||
Name: "Generate",
|
||||
Aliases: []string{"gen"},
|
||||
Usage: "Generates a new copy of P2PRC which can be modified based on your needs",
|
||||
EnvVars: []string{"GENERATE"},
|
||||
Destination: &Generate,
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "ModuleName",
|
||||
Aliases: []string{"mod"},
|
||||
Usage: "New go project module name",
|
||||
EnvVars: []string{"MODULENAME"},
|
||||
Destination: &Modulename,
|
||||
},
|
||||
//--------------------------------
|
||||
}
|
||||
@@ -8,12 +8,7 @@ import (
|
||||
|
||||
var (
|
||||
defaultPath string
|
||||
defaults = map[string]interface{}{
|
||||
"IPTable": "/etc/p2p-rendering/ip_table.json",
|
||||
"DockerContainers": "/home/akilan/Documents/p2prendering/p2p-redering-computation/server/docker/containers/",
|
||||
"DefaultDockerFile": "/home/akilan/Documents/p2prendering/p2p-redering-computation/server/docker/containers/docker-ubuntu-sshd/",
|
||||
"SpeedTestFile":"/etc/p2p-rendering/50.bin",
|
||||
}
|
||||
defaults = map[string]interface{}{}
|
||||
configName = "config"
|
||||
configType = "json"
|
||||
configFile = "config.json"
|
||||
@@ -67,18 +62,30 @@ func Copy(src, dst string) error {
|
||||
return out.Close()
|
||||
}
|
||||
|
||||
// GetPathP2PRC Getting P2PRC Directory from environment variable
|
||||
func GetPathP2PRC()(string,error) {
|
||||
curDir := os.Getenv("P2PRC")
|
||||
return curDir + "/", nil
|
||||
}
|
||||
|
||||
// GetCurrentPath Getting P2PRC Directory from environment variable
|
||||
func GetCurrentPath()(string,error) {
|
||||
curDir := os.Getenv("PWD")
|
||||
return curDir + "/", nil
|
||||
}
|
||||
|
||||
|
||||
// SetDefaults This function to be called only during a
|
||||
// make install
|
||||
func SetDefaults() error {
|
||||
//Getting Current Directory from environment variable
|
||||
curDir := os.Getenv("P2PRC")
|
||||
|
||||
//Setting current directory to default path
|
||||
defaultPath = curDir + "/"
|
||||
defaultPath, err := GetPathP2PRC()
|
||||
if err != nil{
|
||||
return err
|
||||
}
|
||||
|
||||
//Creates ip_table.json in the json directory
|
||||
err := Copy("p2p/ip_table.json","p2p/iptable/ip_table.json")
|
||||
err = Copy("p2p/ip_table.json","p2p/iptable/ip_table.json")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -129,9 +136,11 @@ func SetDefaults() error {
|
||||
|
||||
func ConfigInit()(*Config,error) {
|
||||
|
||||
curDir := os.Getenv("P2PRC")
|
||||
//Setting current directory to default path
|
||||
defaultPath = curDir + "/"
|
||||
defaultPath, err := GetPathP2PRC()
|
||||
if err != nil{
|
||||
return nil, err
|
||||
}
|
||||
//Paths to search for config file
|
||||
configPaths = append(configPaths, defaultPath)
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -17,3 +18,21 @@ func TestSetDefaults(t *testing.T) {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetCurrentPath(t *testing.T) {
|
||||
path, err := GetCurrentPath()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
fmt.Println(path)
|
||||
}
|
||||
|
||||
func TestGetPathP2PRC(t *testing.T) {
|
||||
path, err := GetPathP2PRC()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
fmt.Println(path)
|
||||
}
|
||||
|
||||
201
deploy/LICENSE
201
deploy/LICENSE
@@ -1,201 +0,0 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
148
deploy/README.md
148
deploy/README.md
@@ -1,148 +0,0 @@
|
||||
# Ansible p2p-rendering-computation
|
||||
### Modified from:
|
||||
|
||||
This repository contains an Ansible playbook and instructions to create and manage a single (or many) bare metal deep learning machines. For a description of why Ansible was chosen and what other alternatives were considered, please see [ToolSelection.md](ToolSelection.md)
|
||||
|
||||
## Quick Reference
|
||||
|
||||
If you've already [installed Ansible](#Installation), you can execute the entire playbook by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml
|
||||
```
|
||||
|
||||
You can also execute only the pieces you need by passing tags on the command line:
|
||||
|
||||
- Install only apt/pip [pre-requisites](roles/packages) to execute the other roles:
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "packages"
|
||||
```
|
||||
- Install [Docker CE](roles/docker):
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "docker"
|
||||
```
|
||||
- Install the [Nvidia CUDA GPU drivers](roles/cuda):
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "cuda"
|
||||
```
|
||||
- Install the [Nvidia Docker Runtime](roles/nvidia):
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "nvidia"
|
||||
```
|
||||
|
||||
## What's Included
|
||||
|
||||
After running the ansible script your machines will be loaded with the following:
|
||||
|
||||
1. Docker
|
||||
2. Nvidia CUDA GPU Drivers
|
||||
3. Nvidia Docker Runtime
|
||||
4. TensorFlow GPU Python3 Docker Container
|
||||
5. JupyterLab
|
||||
|
||||
## Using This Repository to Configure Your Environment
|
||||
|
||||
1. [Installation](#Installation)
|
||||
2. [Configuration](#Configuration)
|
||||
3. [Running](#Running)
|
||||
|
||||
---
|
||||
|
||||
### Installation
|
||||
|
||||
Ansible runs on your local machine and sends commands to the remote (machine learning) machines. You'll need ansible installed locally (not on the machine learning boxes).
|
||||
For macOS users, the easiest way to install Ansible is via [Homebrew](https://brew.sh/):
|
||||
|
||||
```bash
|
||||
$ brew install ansible
|
||||
```
|
||||
|
||||
If that's not your cup of tea, install Ansible by following the directions for your machine [here](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-the-control-machine).
|
||||
|
||||
---
|
||||
|
||||
### Configuration
|
||||
|
||||
Gather the following:
|
||||
|
||||
- SSH key or user credentials for the remote account
|
||||
|
||||
**Note:** Ansible does not expose a channel to allow communication between the user and the ssh process to accept a password manually to decrypt an ssh key when using the ssh connection plugin (which is the default). The use of `ssh-agent` is highly recommended.
|
||||
|
||||
- List of servers you wish to manage:
|
||||
- hostnames/IP addresses
|
||||
- SSH port
|
||||
- usernames
|
||||
|
||||
Copy [hosts.example] to `/etc/ansible/hosts` (if it does not already exist). Populate the `hosts` file (no extension) with the information about the servers you gathered above.
|
||||
|
||||
Confirm that you have populated your Ansible hosts file correctly:
|
||||
|
||||
```bash
|
||||
$ ansible-inventory --list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Running
|
||||
|
||||
Once you're satisfied that you correctly populated your `hosts` file, update the `- hosts:` line of [tensorflow.yml] to reflect the hosts or groups you want to configure.
|
||||
|
||||
Examples:
|
||||
|
||||
- Apply against a single host defined as `ml2` in `/etc/ansible/hosts`:
|
||||
```yaml
|
||||
- hosts: ml2
|
||||
```
|
||||
- Apply against a group of hosts defined as `production` in `/etc/ansible/hosts`:
|
||||
```yaml
|
||||
- hosts: production
|
||||
```
|
||||
- Apply against all hosts defined in `/etc/ansible/hosts`:
|
||||
```yaml
|
||||
- hosts: all
|
||||
```
|
||||
|
||||
Then, when you're ready, run the playbook:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --ask-become-pass
|
||||
```
|
||||
|
||||
**Note:** You must have `sudo` access to run the playbook!
|
||||
|
||||
Review the output:
|
||||
|
||||
- `[ok]` means no change (this task was already completed)
|
||||
- `[changed]` means the task successfully ran and the change was applied
|
||||
- `[unreachable]` means the host could not be reached
|
||||
- `[failed]` means the task ran but failed to complete
|
||||
|
||||
`[ok]` and `[changed]` are successful outcomes. Any `[unreachable]` and `[failed]` outputs should be investigated and resolved.
|
||||
|
||||
**Note:** This Ansible playbook is idempotent; once a configuration has been successfully applied, if you apply it again, all actions will report `[ok]`.
|
||||
|
||||
## Executing Tensorflow Jobs in Your New Environment
|
||||
|
||||
1. Point your browser to http://<hostname>:8888 and login with the password you provided.
|
||||
2. The `jupyter.volumes.source` folder will be mounted as the `notebooks` folder.
|
||||
3. Edit and execute your Jupyter notebooks as normal!
|
||||
|
||||
### Command Line Access
|
||||
|
||||
If you need to drop into a GPU-powered TensorFlow environment, SSH into the remote machine and execute the following:
|
||||
|
||||
```bash
|
||||
$ docker run --runtime=nvidia -it --rm tensorflow/tensorflow:latest-gpu-py3 bash
|
||||
```
|
||||
|
||||
**Note:** You must be a member of the `docker` group or have `sudo` access on the _remote machine_ to execute docker commands.
|
||||
|
||||
---
|
||||
|
||||
## Additional Files
|
||||
|
||||
- [ansible.cfg](ansible.cfg) enables SSH credential forwarding. This is a necessary step during data synchronization, as Ansible delegates those credentials to the master/writer host to push the data folder out to each of the mirrors.
|
||||
- [Dockerfile](Dockerfile) is used to build the Arricor TensorFlow image. See [Docker.md](Docker.md) for additional details.
|
||||
- [hosts.example](hosts.example) is an example of the Ansible inventory hosts file saved in `/etc/ansible/hosts`
|
||||
- [secrets.example.yml](secrets.example.yml) is an example of the expected structure of the `secrets.yml` file
|
||||
@@ -1 +0,0 @@
|
||||
Test if it's detected
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
all:
|
||||
vars:
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
main:
|
||||
hosts:
|
||||
host1:
|
||||
ansible_host: 0.0.0.0 # Replace with your remote IP
|
||||
ansible_port: 44003 # Replace with your remote SSH port
|
||||
ansible_user: master # Replace wtih your username
|
||||
ansible_ssh_pass: password
|
||||
ansible_sudo_pass: password
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
|
||||
- hosts: all
|
||||
|
||||
tasks:
|
||||
- name: simple-ansibleplaybook
|
||||
debug:
|
||||
msg: Your are running 'simple-ansibleplaybook' example
|
||||
@@ -1,93 +0,0 @@
|
||||
# Configuration Management Tool Selection
|
||||
|
||||
## Summary
|
||||
|
||||
Ansible is proposed as the configuration management tool for this project due to the following characteristics:
|
||||
|
||||
- no agent installation on the managed nodes
|
||||
- push-based configuration changes do not require a separate machine to be provisioned
|
||||
- commands can be pushed from any developer/operator workstation
|
||||
- all updates are performed over SSH
|
||||
|
||||
## Tools Considered
|
||||
|
||||
Where a tool offers an open source and an enterprise version (e.g., Puppet, Terraform), only the open source version was considered. Web or other graphical user interfaces (GUIs) were not considered, as the goal of this phase is to produce infrastructure as code which can be run (and re-run) from the command line interface (CLI).
|
||||
|
||||
- [Ansible](#Ansible)
|
||||
- [Chef](#Chef)
|
||||
- [Puppet](#Puppet)
|
||||
- [Salt](#Salt)
|
||||
- [Terraform](#Terraform)
|
||||
|
||||
### Ansible
|
||||
|
||||
[Ansible's simple interface and usability fit right into the sys admin mindset, and in a shop with lots of Linux and Unix systems, Ansible is quick and easy to run right out of the gate](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-Salt.html?page=4).
|
||||
|
||||
#### Ansible Pros
|
||||
|
||||
- Requires no agent installation on the managed nodes.
|
||||
- Push-based.
|
||||
- Does not require a separate machine to be provisioned. Commands can be pushed from any developer/operator workstation.
|
||||
- All updates are performed over SSH.
|
||||
|
||||
#### Ansible Cons
|
||||
|
||||
- Push-based synchronization means that the configuration is only refreshed on demand.
|
||||
|
||||
### Chef
|
||||
|
||||
[Chef has a stable and well-designed layout, and while it's not quite up to the level of Puppet in terms of raw features, it's a very capable solution. Chef may pose the most difficult learning curve to administrators who lack significant programming experience, but it could be the most natural fit for development-minded admins and development shops](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-Salt.html?page=4).
|
||||
|
||||
#### Chef Pros
|
||||
|
||||
- AWS OpsWorks is based on Chef; any future migration to AWS could benefit from the existing configuration files.
|
||||
|
||||
#### Chef Cons
|
||||
|
||||
- **(Fatal)** Requires a master server and a workstation that will need to be maintained separately from the already provisioned deep learning machines.
|
||||
- **(Fatal)** Requires an agent (client) to be installed on each machine.
|
||||
- Requires an on-premises Chef server and [Push Jobs](https://docs.chef.io/push_jobs.html) plugin to push configurations manually. Otherwise, managed machines must check in with the master on a schedule.
|
||||
|
||||
### Puppet
|
||||
|
||||
[Puppet is the most mature and probably the most approachable of the four from a usability standpoint, though a solid knowledge of Ruby is highly recommended. Puppet is not as streamlined as Ansible or Salt, and its configuration can get Byzantine at times. Puppet is the safest bet for heterogeneous environments, but you may find Ansible or Salt to be a better fit in a larger or more homogenous infrastructure](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html?page=4).
|
||||
|
||||
#### Puppet Pros
|
||||
|
||||
- Configuration can be kept up to date via a schedule or manual configuration pushes.
|
||||
|
||||
#### Puppet Cons
|
||||
|
||||
- **(Fatal)** Requires a "puppet master" (server) that will need to be maintained separately from the already provisioned machines.
|
||||
- **(Fatal)** Requires an agent (client) to be installed on each machine.
|
||||
|
||||
### Salt
|
||||
|
||||
[Salt is the sleekest and most robust of the four, and like Ansible it will resonate with sys admins. Highly scalable and quite capable, Salt is hamstrung only by the Web UI](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html?page=4).
|
||||
|
||||
#### Salt Pros
|
||||
|
||||
- Push-based.
|
||||
- Can function via regular SSH or via client agents called "minions".
|
||||
|
||||
#### Salt Cons
|
||||
|
||||
- When running agentless, push-based synchronization means that the configuration is only refreshed on demand.
|
||||
|
||||
### Terraform
|
||||
|
||||
The initial hope was that [Terraform](https://terraform.io) could be used for configuration management for this project. However, after reviewing the [available Terraform provider plugins](https://terraform.io/docs/providers) and other available material, it was agreed on 23 January 2019 that Terraform was not suitable for this stage of the project.
|
||||
|
||||
In the future, if Arricor migrates its instances to a cloud service provider (CSP) such as AWS, Azure, or Google Cloud Platform, this configuration may be reusable as a first step via the [terraform-provisioner-ansible](https://github.com/jonmorehouse/terraform-provisioner-ansible) plugin. This plugin allows the user to create CSP instances, then execute the given Ansible playbook against those instances.
|
||||
|
||||
#### Terraform Pros
|
||||
|
||||
- Cloud native with support for multiple service providers.
|
||||
|
||||
#### Terraform Cons
|
||||
|
||||
- **(Fatal)** No support for provisioning bare metal instances.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Ansible allows for management of a server from scratch, with only the requirement to have a valid username and password. Additionally, its agentless approach avoids having to configure and maintain an admin server, further simplifying operations at the scale Arricor requires.
|
||||
@@ -1,5 +0,0 @@
|
||||
[defaults]
|
||||
host_key_checking = False
|
||||
|
||||
[ssh_connection]
|
||||
ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s
|
||||
@@ -1,25 +0,0 @@
|
||||
---
|
||||
all:
|
||||
vars:
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
test:
|
||||
hosts:
|
||||
test1:
|
||||
ansible_host: 127.0.0.1 # Replace with your remote IP
|
||||
ansible_port: 2222 # Replace with your remote SSH port
|
||||
ansible_user: vagrant # Replace wtih your username
|
||||
|
||||
production:
|
||||
hosts:
|
||||
writer:
|
||||
ansible_host: 192.168.1.100 # Replace with your remote IP
|
||||
ansible_port: 2202 # Replace with your remote SSH port
|
||||
ansible_user: ansible # Replace wtih your username
|
||||
mirror1:
|
||||
ansible_host: 192.168.1.101 # Replace with your remote IP
|
||||
ansible_port: 2202 # Replace with your remote SSH port
|
||||
ansible_user: ansible # Replace wtih your username
|
||||
mirror2:
|
||||
ansible_host: 192.168.1.102 # Replace with your remote IP
|
||||
ansible_port: 2202 # Replace with your remote SSH port
|
||||
ansible_user: ansible # Replace wtih your username
|
||||
@@ -1,38 +0,0 @@
|
||||
################################################################################
|
||||
# This file consists of three main sections:
|
||||
# 1. vars - variables used throughout the playbook are defined here. If you want to change packages or version, this is the place to do it.
|
||||
# 2. tasks - these are the steps that will be executed by the playbook. This is a straightforward, sequential playbook.
|
||||
# 3. handlers - these steps run only in response to specific events. If the event does not occur, the handler does not run.
|
||||
################################################################################
|
||||
---
|
||||
- hosts: test # Alternatively, 'canary', 'blue'|'green', 'all' according to your deployment strategy and your /etc/ansible/hosts file
|
||||
become: yes # All commands must run as root
|
||||
strategy: free # Allows all hosts to run to the end of the playbook as fast as they can
|
||||
|
||||
vars_files:
|
||||
- secrets.yml
|
||||
|
||||
roles:
|
||||
- role: packages # General Dependencies
|
||||
tags: [packages]
|
||||
|
||||
# - role: rsync # Rsync Setup
|
||||
# tags: [rsync]
|
||||
# vars:
|
||||
# sync: "{{ secrets.sync }}"
|
||||
|
||||
- role: docker # General Docker setup
|
||||
tags: [docker]
|
||||
vars:
|
||||
users: "{{ secrets.users }}"
|
||||
|
||||
- role: cuda # CUDA Driver Setup
|
||||
tags: [cuda]
|
||||
|
||||
- role: nvidia # NVIDIA-Docker Runtime Setup
|
||||
tags: [nvidia]
|
||||
|
||||
- role: jupyter # Launch Jupyter Container in Docker
|
||||
tags: [jupyter]
|
||||
vars:
|
||||
volumes: "{{ secrets.volumes }}"
|
||||
@@ -1,23 +0,0 @@
|
||||
# The `cuda` Role
|
||||
|
||||
The `cuda` role installs the Nvidia CUDA GPU repository and drivers.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "cuda"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Apt Repositories (Debian/Ubuntu)
|
||||
|
||||
- Nvidia CUDA
|
||||
|
||||
### Apt GPG Signing Keys (Debian/Ubuntu)
|
||||
|
||||
- cudatools <cudatools@nvidia.com>
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- cuda v10.x
|
||||
Binary file not shown.
@@ -1,5 +0,0 @@
|
||||
---
|
||||
# CUDA Driver Setup
|
||||
- name: reboot the machine
|
||||
reboot:
|
||||
listen: "reboot the machine"
|
||||
@@ -1,22 +0,0 @@
|
||||
---
|
||||
# Cuda Driver Package Installation
|
||||
- name: copy the CUDA repo onto the machine
|
||||
copy:
|
||||
src: "{{ cuda.apt.repo }}"
|
||||
dest: "{{ general.working_dir }}/{{ cuda.apt.repo }}"
|
||||
|
||||
- name: get the CUDA signing key
|
||||
apt_key:
|
||||
state: present
|
||||
url: "{{ cuda.apt.signing_key_url }}"
|
||||
|
||||
- name: add the CUDA repo
|
||||
apt:
|
||||
deb: "{{ general.working_dir }}/{{ cuda.apt.repo }}"
|
||||
state: present
|
||||
|
||||
- name: install the CUDA drivers
|
||||
apt:
|
||||
name: "{{ cuda.apt.package }}"
|
||||
update_cache: yes
|
||||
notify: "reboot the machine"
|
||||
@@ -1,3 +0,0 @@
|
||||
---
|
||||
- import_tasks: debian.yml
|
||||
when: ansible_facts['os_family']|lower == 'debian' # debian, ubuntu
|
||||
@@ -1,9 +0,0 @@
|
||||
---
|
||||
cuda:
|
||||
apt:
|
||||
package: cuda=10.* # known good with 10.0.130-1
|
||||
repo: cuda-repo-ubuntu1804_10.0.130-1_amd64.deb # Network install
|
||||
signing_key_url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
|
||||
|
||||
general:
|
||||
working_dir: /tmp
|
||||
@@ -1,33 +0,0 @@
|
||||
# The `docker` Role
|
||||
|
||||
The `docker` role installs the Docker CE repository and runtime packages.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "docker"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Apt Repositories (Debian/Ubuntu)
|
||||
|
||||
- Docker CE
|
||||
|
||||
### Apt GPG Signing Keys (Debian/Ubuntu)
|
||||
|
||||
- Docker Release (CE deb) <docker@docker.com>
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- docker-ce v5:18.x
|
||||
|
||||
### System Services
|
||||
|
||||
- `docker` (enabled and started)
|
||||
|
||||
### User Groups
|
||||
|
||||
- docker
|
||||
|
||||
All users listed under `secrets.users` in `secrets.yml` will be added to the `docker` group.
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
# Docker Package Installation
|
||||
- name: add Docker's official GPG key
|
||||
apt_key:
|
||||
state: present
|
||||
url: "{{ docker.apt.signing_key_url }}"
|
||||
|
||||
- name: add Docker apt repository
|
||||
apt_repository:
|
||||
repo: "{{ docker.apt.repo }}"
|
||||
state: present
|
||||
|
||||
- name: install Docker CE
|
||||
apt:
|
||||
name: "{{ docker.apt.package }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
@@ -1,23 +0,0 @@
|
||||
---
|
||||
# General Docker Setup
|
||||
- import_tasks: debian.yml
|
||||
when: ansible_facts['os_family']|lower == 'debian' # debian, ubuntu
|
||||
|
||||
- name: ensure the docker service starts at boot
|
||||
systemd:
|
||||
name: "{{ docker.service_name }}"
|
||||
enabled: yes
|
||||
state: started
|
||||
|
||||
- name: create Docker group to avoid running as root
|
||||
group:
|
||||
name: "{{ docker.group_name }}"
|
||||
state: present
|
||||
|
||||
- name: add users to Docker group
|
||||
user:
|
||||
append: yes
|
||||
groups:
|
||||
- "{{ docker.group_name }}"
|
||||
name: "{{ item }}"
|
||||
loop: "{{ users }}"
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
docker:
|
||||
apt:
|
||||
package: docker-ce=5:18.* # known good with 5:18.09.1~3-0~ubuntu-bionic
|
||||
repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
|
||||
signing_key_url: https://download.docker.com/linux/ubuntu/gpg
|
||||
group_name: docker
|
||||
service_name: docker
|
||||
@@ -1,31 +0,0 @@
|
||||
# The `nvidia` Role
|
||||
|
||||
The `nvidia` role installs the Nvidia Docker runtime, enabling access to Nvidia CUDA GPU hardware from within Docker containers.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "nvidia"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Apt Repositories (Debian/Ubuntu)
|
||||
|
||||
- libnvidia_container
|
||||
- nvidia_container_runtime
|
||||
- nvidia_docker
|
||||
|
||||
### Apt GPG Signing Keys (Debian/Ubuntu)
|
||||
|
||||
- NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- nvidia-docker2 v2.x
|
||||
|
||||
Containers are launched with the Nvidia Docker runtime by passing the `--runtime=nvidia` flag, e.g.,
|
||||
|
||||
```bash
|
||||
$ docker run --rm --runtime=nvidia nvidia/cuda nvidia-smi
|
||||
```
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
- name: reload the docker service
|
||||
systemd:
|
||||
name: docker
|
||||
state: reloaded
|
||||
listen: "reload the docker service"
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
# Nvidia Docker Runtime Installation
|
||||
- name: get the NVIDIA signing key
|
||||
apt_key:
|
||||
state: present
|
||||
url: "{{ nvidia.apt.signing_key_url }}"
|
||||
|
||||
- name: add the libnvidia-container repo
|
||||
apt_repository:
|
||||
filename: "{{ nvidia.apt.repo_filename }}"
|
||||
repo: "{{ nvidia.apt.repos.libnvidia_container }}"
|
||||
state: present
|
||||
|
||||
- name: add the nvidia-container-runtime repo
|
||||
apt_repository:
|
||||
filename: "{{ nvidia.apt.repo_filename }}"
|
||||
repo: "{{ nvidia.apt.repos.nvidia_container_runtime }}"
|
||||
state: present
|
||||
|
||||
- name: add the nvidia-docker repo
|
||||
apt_repository:
|
||||
filename: "{{ nvidia.apt.repo_filename }}"
|
||||
repo: "{{ nvidia.apt.repos.nvidia_docker }}"
|
||||
state: present
|
||||
|
||||
- name: install the nvidia-docker2 runtime
|
||||
apt:
|
||||
name: "{{ nvidia.apt.package }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
notify: "reload the docker service"
|
||||
@@ -1,4 +0,0 @@
|
||||
---
|
||||
# Nvidia Docker Runtime Setup
|
||||
- import_tasks: debian.yml
|
||||
when: ansible_facts['os_family']|lower == 'debian' # debian, ubuntu
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
nvidia:
|
||||
apt:
|
||||
package: nvidia-docker2=2.* # known good with 2.0.3+docker18.09.1-1
|
||||
repo_filename: nvidia_github_io_nvidia_docker_ubuntu18_04_nvidia_docker.list
|
||||
repos:
|
||||
libnvidia_container: deb https://nvidia.github.io/libnvidia-container/ubuntu18.04/$(ARCH) /
|
||||
nvidia_container_runtime: deb https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/$(ARCH) /
|
||||
nvidia_docker: deb https://nvidia.github.io/nvidia-docker/ubuntu18.04/$(ARCH) /
|
||||
signing_key_url: https://nvidia.github.io/nvidia-docker/gpgkey
|
||||
@@ -1,26 +0,0 @@
|
||||
# The `packages` Role
|
||||
|
||||
The `packages` role installs the operating system and Python pre-requisites that are required to execute the other roles.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "packages"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- apt-transport-https v1.x
|
||||
- ca-certificates v20180409
|
||||
- curl v7.x
|
||||
- gnupg-agent v2.x
|
||||
- python-apt v1.x
|
||||
- python-pip v9.x
|
||||
- python-setuptools v39.x
|
||||
- software-properties-common v0.96.24.32.7
|
||||
|
||||
### Python (pip) Packages
|
||||
|
||||
- docker v3.7.0 or greater
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
# General Dependencies
|
||||
- name: install general apt dependencies
|
||||
apt:
|
||||
name: "{{ general.dependencies.apt }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
|
||||
- name: install general pip dependencies for Ansible
|
||||
pip:
|
||||
name: "{{ general.dependencies.pip }}"
|
||||
state: present
|
||||
@@ -1,3 +0,0 @@
|
||||
---
|
||||
- import_tasks: debian.yml
|
||||
when: ansible_facts['os_family']|lower == 'debian' # debian, ubuntu
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
# General Dependencies
|
||||
general:
|
||||
dependencies:
|
||||
apt:
|
||||
- apt-transport-https=1.* # known good with 1.6.6ubuntu0.1 all
|
||||
- ca-certificates=20180409 # known good with 20180409
|
||||
- curl=7.* # known good with 7.58.0-2ubuntu3.5
|
||||
- gnupg-agent=2.* # known good with 2.2.4-1ubuntu1.2
|
||||
- python3-apt=1.* # known good with 1.6.3
|
||||
- python3-pip=9.* # known good with 9.0.1-2.3~ubuntu1
|
||||
- python3-setuptools=39.* # known good with 39.0.1-2
|
||||
- software-properties-common=0.96.24.32.7 # known good with 0.96.24.32.7
|
||||
- git
|
||||
- golang
|
||||
pip:
|
||||
- docker>=3.7.0 # known good with 3.7.0
|
||||
@@ -1,16 +0,0 @@
|
||||
motd file = /etc/rsyncd.motd
|
||||
log file = /var/log/rsyncd.log
|
||||
pid file = /var/run/rsyncd.pid
|
||||
lock file = /var/run/rsync.lock
|
||||
|
||||
[jupyter_data]
|
||||
path = /nfs/jupyter/data
|
||||
comment = Jupyter data files
|
||||
uid = nobody
|
||||
gid = nobody
|
||||
read only = yes
|
||||
list = yes
|
||||
auth users = @rsync:ro
|
||||
secrets file = /etc/rsyncd.secrets
|
||||
hosts allow = 192.168.1.0/24
|
||||
hosts deny = 0.0.0.0/0
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
- name: restart the rsync service
|
||||
systemd:
|
||||
name: "{{ rsync.service_name }}"
|
||||
state: restarted
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
# Configure the rsync daemon on the master
|
||||
- name: copy rsync.conf up to the server
|
||||
copy:
|
||||
src: "{{ rsync.config_file.src }}"
|
||||
dest: "{{ rsync.config_file.dest }}"
|
||||
notify:
|
||||
- restart the rsync service
|
||||
when: inventory_hostname == rsync.master
|
||||
|
||||
- name: configure rsync on the server
|
||||
service:
|
||||
enabled: yes
|
||||
name: "{{ rsync.service_name }}"
|
||||
state: started
|
||||
when: inventory_hostname == rsync.master
|
||||
|
||||
# Synchronize the data deploy across hosts
|
||||
- name: synchronize the data deploy
|
||||
delegate_to: "{{ rsync.master }}"
|
||||
synchronize:
|
||||
src: "{{ sync.source }}"
|
||||
dest: "{{ sync.destination }}"
|
||||
rsync_opts: "{{ rsync.options }}"
|
||||
use_ssh_args: yes
|
||||
when: inventory_hostname != rsync.master
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
rsync:
|
||||
config_file:
|
||||
src: rsyncd.conf
|
||||
dest: /etc/rsyncd.conf
|
||||
master: ml1
|
||||
options:
|
||||
- "-e ssh"
|
||||
- "--no-motd"
|
||||
service_name: rsync
|
||||
@@ -1,15 +0,0 @@
|
||||
################################################################################
|
||||
# This file contains secrets that should not be checked into VCS.
|
||||
################################################################################
|
||||
---
|
||||
secrets:
|
||||
# Cachux7I
|
||||
password_hash: sha1:b9bbcb5e92ad:0358d2ce7c34192afa50fc5e5143ed91d75eda0d
|
||||
sync:
|
||||
source: /nfs/data
|
||||
destination: /nfs
|
||||
users:
|
||||
- ansible
|
||||
volumes:
|
||||
data: /nfs/data:/data
|
||||
source: /nfs/src:/tf
|
||||
386
generate/generate.go
Normal file
386
generate/generate.go
Normal file
@@ -0,0 +1,386 @@
|
||||
// Package generate The purpose of this package is to ensure that we can extend the use-case of P2PRC.
|
||||
// We will create a project directory with the template to extend the use-case of P2PRC
|
||||
package generate
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/config"
|
||||
"github.com/otiai10/copy"
|
||||
"go/ast"
|
||||
"go/token"
|
||||
modfile "golang.org/x/mod/modfile"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// NewProject Struct information required when creating a new project
|
||||
type NewProject struct {
|
||||
Name string
|
||||
Module string
|
||||
NewDir string
|
||||
P2PRCPath string
|
||||
CurrentModule string
|
||||
Option *copy.Options
|
||||
Token *token.FileSet
|
||||
AST *ast.File
|
||||
FileNameAST string
|
||||
}
|
||||
|
||||
// GenerateNewProject creates a new copy of the P2PRC
|
||||
// project for custom modification
|
||||
func GenerateNewProject(name string, module string) error {
|
||||
// Create new variable of type NewProject
|
||||
var newProject NewProject
|
||||
//Setting module name to the new project
|
||||
newProject.Module = module
|
||||
// Get path of the current directory
|
||||
curDir, err := config.GetCurrentPath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Folder name of the new generated project
|
||||
newProject.NewDir = curDir + name + "/"
|
||||
// Create a new folder based on name entered
|
||||
err = CreateFolder(name, curDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// get path of P2PRC
|
||||
P2PRCPATH, err := config.GetPathP2PRC()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Assign P2PRC path to the newly generated project
|
||||
newProject.P2PRCPath = P2PRCPATH
|
||||
// Steps:
|
||||
// - copy all files from P2PRC
|
||||
// - remove go.mod and go.sum and create new ones
|
||||
|
||||
// Files we require to skip
|
||||
|
||||
var Options copy.Options
|
||||
|
||||
// IF YOU ARE RELEASING AN EXTENSION OF P2PRC
|
||||
// MODIFY HERE (AN EXTENSION FROM YOUR EXTENSION :P)
|
||||
// Skip or have the appropriate files and directories not needed
|
||||
//----------------------------------------------------------------
|
||||
// Action performed:
|
||||
// - Ensuring main.go file exists
|
||||
// - Ensuring generate.go file exists
|
||||
// - Ensuring modifyGenerate.go file exists
|
||||
// - Ensuring generate_test.go file exists
|
||||
// - Ensuring server/server.go file exists
|
||||
// - Ensuring server/gopsutil.go file exists
|
||||
// - Ensuring server/gpu.go file exists
|
||||
// - Ensuring cmd/action.go file exists
|
||||
// - Ensuring cmd/flags.go file exists
|
||||
// - Skipping all .go files apart from the ones listed above
|
||||
// - Skipping go.mod file
|
||||
// - Skipping go.sum file
|
||||
// - Skipping .idea/ directory
|
||||
// - Skipping Makefile file
|
||||
// - Skipping <Project Name>/ directory
|
||||
//----------------------------------------------------------------
|
||||
Options.Skip = func(src string) (bool, error) {
|
||||
switch {
|
||||
case strings.HasSuffix(src, "main.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "generate.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "modifyGenerate.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "generate_test.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "server/server.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "server/gopsutil.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "server/gpu.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "cmd/action.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "cmd/flags.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "config/config.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, "config/config_test.go"):
|
||||
return false, nil
|
||||
case strings.HasSuffix(src, ".go"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, "go.mod"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, "go.sum"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, ".idea"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, "Makefile"):
|
||||
return true, nil
|
||||
case strings.HasSuffix(src, name):
|
||||
return true, nil
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Storing type option in the struct new project
|
||||
newProject.Option = &Options
|
||||
|
||||
// Copies all files from P2PRC to the new project created
|
||||
err = copy.Copy(newProject.P2PRCPath, newProject.NewDir, *newProject.Option)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Creating a new go.mod file in the appropriate directory
|
||||
err = newProject.CreateGoMod()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get current project mod name
|
||||
err = newProject.GetCurrentGoModule()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Change the appropriate imports
|
||||
err = newProject.ChangeImportFiles()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Add changes inside the new project
|
||||
err = newProject.GitAdd()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// commit changes inside the new project
|
||||
err = newProject.GitCommit()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Creates go.sum file
|
||||
//err = newProject.CreateGoModTidy()
|
||||
//if err != nil {
|
||||
// return err
|
||||
//}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateFolder Creates a new folder based on the name and path provided
|
||||
func CreateFolder(name string, path string) error {
|
||||
//Create a folder/directory at a full qualified path
|
||||
err := os.Mkdir(path+name, 0755)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateGoMod Creates a new go module for the new project created
|
||||
func (a *NewProject) CreateGoMod() error {
|
||||
// Create new go.mod in the appropriate directory
|
||||
cmd := exec.Command("go", "mod", "init", a.Module)
|
||||
cmd.Dir = a.NewDir
|
||||
if err := cmd.Run(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *NewProject) CreateGoModTidy() error {
|
||||
// Installs all the appropriate dependencies and creates go.sum
|
||||
cmd := exec.Command("go", "mod", "tidy")
|
||||
cmd.Dir = a.NewDir
|
||||
if err := cmd.Run(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ChangeImportFiles Changes Appropriate imports in the appropriate file
|
||||
func (a *NewProject) ChangeImportFiles() error {
|
||||
// IF YOU ARE RELEASING AN EXTENSION OF P2PRC
|
||||
// MODIFY HERE (AN EXTENSION FROM YOUR EXTENSION :P)
|
||||
//----------------------------------------------------------------
|
||||
// Action performed:
|
||||
// Files we would need to modify the imports in
|
||||
// - generate/generate.go -> config module
|
||||
// - generate/generate_test.go -> config module
|
||||
// - cmd/action.go -> config module, server module, generate module
|
||||
// - cmd/flags.go -> config module, server module, generate module
|
||||
// - server/server.go -> config module
|
||||
// - main.go -> cmd module
|
||||
//-----------------------------------------------------------------
|
||||
|
||||
//-----------------------------------------------------------------
|
||||
// 1.0 - generate/generate.go -> config module
|
||||
a.FileNameAST = a.NewDir + "generate/generate.go"
|
||||
// Get AST information of the file
|
||||
err := a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports(a.CurrentModule+"/config", a.Module+"/config")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
//-----------------------------------------------------------------
|
||||
// 1.1 - generate/generate_test.go -> config module
|
||||
a.FileNameAST = a.NewDir + "generate/generate_test.go"
|
||||
// Get AST information of the file
|
||||
err = a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports(a.CurrentModule+"/config", a.Module+"/config")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
//-----------------------------------------------------------------
|
||||
//-----------------------------------------------------------------
|
||||
// 2.0 - cmd/action.go -> config module, server module, generate module
|
||||
a.FileNameAST = a.NewDir + "cmd/action.go"
|
||||
// Get AST information of the file
|
||||
err = a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports(a.CurrentModule+"/config", a.Module+"/config")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
err = a.ChangeImports(a.CurrentModule+"/server", a.Module+"/server")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
err = a.ChangeImports(a.CurrentModule+"/generate", a.Module+"/generate")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
//-----------------------------------------------------------------
|
||||
// 2.1 - cmd/flags.go -> config module, server module, generate module
|
||||
a.FileNameAST = a.NewDir + "cmd/flags.go"
|
||||
// Get AST information of the file
|
||||
err = a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports(a.CurrentModule+"/config", a.Module+"/config")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
err = a.ChangeImports(a.CurrentModule+"/server", a.Module+"/server")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
err = a.ChangeImports(a.CurrentModule+"/generate", a.Module+"/generate")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
//-----------------------------------------------------------------
|
||||
//-----------------------------------------------------------------
|
||||
// 3.0 - server/server.go -> config module
|
||||
a.FileNameAST = a.NewDir + "server/server.go"
|
||||
// Get AST information of the file
|
||||
err = a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports(a.CurrentModule+"/config", a.Module+"/config")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
//-----------------------------------------------------------------
|
||||
//-----------------------------------------------------------------
|
||||
// 4.0 - server/server.go -> config module
|
||||
a.FileNameAST = a.NewDir + "main.go"
|
||||
// Get AST information of the file
|
||||
err = a.GetASTGoFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Change the appropriate Go file
|
||||
err = a.ChangeImports(a.CurrentModule+"/cmd", a.Module+"/cmd")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Writes the change to the appropriate file
|
||||
err = a.WriteGoAst()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
//-----------------------------------------------------------------
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetCurrentGoModule Gets the current go module name
|
||||
func (a *NewProject) GetCurrentGoModule() error {
|
||||
goModBytes, err := ioutil.ReadFile(a.P2PRCPath + "go.mod")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
modName := modfile.ModulePath(goModBytes)
|
||||
fmt.Fprintf(os.Stdout, "modName=%+v\n", modName)
|
||||
|
||||
// Set current module to struct of file NewProject
|
||||
a.CurrentModule = modName
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *NewProject) GitAdd() error {
|
||||
// Installs all the appropriate dependencies and creates go.sum
|
||||
cmd := exec.Command("git", "add", ".")
|
||||
cmd.Dir = a.NewDir
|
||||
if err := cmd.Run(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *NewProject) GitCommit() error {
|
||||
// Installs all the appropriate dependencies and creates go.sum
|
||||
cmd := exec.Command("git", "commit", "-m=removed appropriate go files")
|
||||
cmd.Dir = a.NewDir
|
||||
if err := cmd.Run(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
112
generate/generate_test.go
Normal file
112
generate/generate_test.go
Normal file
@@ -0,0 +1,112 @@
|
||||
package generate
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"git.sr.ht/~akilan1999/p2p-rendering-computation/config"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Tests the create folder function creates a folder
|
||||
// This test will create a folder in the temporary
|
||||
// directory
|
||||
func TestCreateFolder(t *testing.T) {
|
||||
err := CreateFolder("test", "/tmp/")
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Testing if a new project is created successfully
|
||||
func TestGenerateNewProject(t *testing.T) {
|
||||
// Checking if a new project is created successfully
|
||||
err := GenerateNewProject("p2prctest","p2prctest")
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Testing AST function to ensure imports are
|
||||
// working as intended
|
||||
func TestChangingImportAST(t *testing.T) {
|
||||
// Create a new variable of type NewProject
|
||||
var np NewProject
|
||||
// Get current directory
|
||||
path, err := config.GetCurrentPath()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
// Create testcase scenario
|
||||
err = config.Copy(path + "testcaseAST.go", path + "/Test/testcaseAST.go")
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
// Sets new directory to the folder test
|
||||
np.NewDir = path + "Test/"
|
||||
// Sets file name to be opened and modified in the AST
|
||||
np.FileNameAST = path + "Test/" + "testcaseAST.go"
|
||||
// Call the Read AST function
|
||||
err = np.GetASTGoFile()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
// Change an import
|
||||
err = np.ChangeImports("fmt", "lolol")
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
// Write those saved changes
|
||||
err = np.WriteGoAst()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// Testing the if Go Mod is created
|
||||
func TestNewProject_CreateGoMod(t *testing.T) {
|
||||
// Create a new variable of type NewProject
|
||||
var np NewProject
|
||||
path, err := config.GetCurrentPath()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
// Set new project name as Test
|
||||
np.Name = "Test"
|
||||
// Set new project module as github.com/Test
|
||||
np.Module = "github.com/Test"
|
||||
// Set Path of the new project
|
||||
np.NewDir = path + "Test/"
|
||||
// Creating a go.mod file
|
||||
err = np.CreateGoMod()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Testing if the current go module is returned
|
||||
func TestNewProject_GetCurrentGoModule(t *testing.T) {
|
||||
// Create a new variable of type NewProject
|
||||
var np NewProject
|
||||
path, err := config.GetPathP2PRC()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
// Set Current project path
|
||||
np.P2PRCPath = path
|
||||
// Get module name
|
||||
err = np.GetCurrentGoModule()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
t.Error(err)
|
||||
}
|
||||
fmt.Println(np.CurrentModule)
|
||||
}
|
||||
49
generate/modifyGenerate.go
Normal file
49
generate/modifyGenerate.go
Normal file
@@ -0,0 +1,49 @@
|
||||
package generate
|
||||
|
||||
import (
|
||||
"go/parser"
|
||||
"go/printer"
|
||||
"go/token"
|
||||
"os"
|
||||
)
|
||||
|
||||
|
||||
// GetASTGoFile Gets AST of the Go file provided
|
||||
func (np *NewProject)GetASTGoFile() error{
|
||||
fset := token.NewFileSet()
|
||||
node, err := parser.ParseFile(fset, np.FileNameAST, nil, parser.ParseComments)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
//Write Token information to the struct
|
||||
np.Token = fset
|
||||
// Write AST information the struct
|
||||
np.AST = node
|
||||
return nil
|
||||
}
|
||||
|
||||
// ChangeImports Changes import of the AST
|
||||
func (np *NewProject)ChangeImports(CurrentImport string,ChangedImport string) error {
|
||||
// Iterating through the loop and changing the appropriate import
|
||||
for i, spec := range np.AST.Imports {
|
||||
// If the current import is found then change it
|
||||
if spec.Path.Value == "\"" + CurrentImport + "\"" {
|
||||
np.AST.Imports[i].Path.Value = "\"" + ChangedImport + "\""
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// WriteGoAst Write changed imports back to the AST
|
||||
func (np *NewProject)WriteGoAst() error {
|
||||
// write new AST to file
|
||||
f, err := os.Create(np.FileNameAST)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer f.Close()
|
||||
if err := printer.Fprint(f, np.Token, np.AST); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
9
generate/testcaseAST.go
Normal file
9
generate/testcaseAST.go
Normal file
@@ -0,0 +1,9 @@
|
||||
package generate
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
func TestCaseAST() {
|
||||
fmt.Println("lol")
|
||||
}
|
||||
4
go.mod
4
go.mod
@@ -18,17 +18,19 @@ require (
|
||||
github.com/moby/sys/mount v0.2.0 // indirect
|
||||
github.com/moby/term v0.0.0-20201110203204-bea5bbe245bf // indirect
|
||||
github.com/morikuni/aec v1.0.0 // indirect
|
||||
github.com/otiai10/copy v1.6.0
|
||||
github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2
|
||||
github.com/shirou/gopsutil v3.20.12+incompatible
|
||||
github.com/spf13/viper v1.4.0
|
||||
github.com/urfave/cli/v2 v2.3.0
|
||||
gitlab.com/NebulousLabs/fastrand v0.0.0-20181126182046-603482d69e40 // indirect
|
||||
gitlab.com/NebulousLabs/go-upnp v0.0.0-20181011194642-3a71999ed0d3
|
||||
golang.org/x/mod v0.3.0
|
||||
golang.org/x/net v0.0.0-20210119194325-5f4716e94777 // indirect
|
||||
golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4 // indirect
|
||||
golang.org/x/text v0.3.5 // indirect
|
||||
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba // indirect
|
||||
google.golang.org/grpc v1.35.0 // indirect
|
||||
gopkg.in/yaml.v2 v2.4.0 // indirect
|
||||
gopkg.in/yaml.v2 v2.4.0
|
||||
gotest.tools/v3 v3.0.3 // indirect
|
||||
)
|
||||
|
||||
9
go.sum
9
go.sum
@@ -462,6 +462,14 @@ github.com/opencontainers/runtime-spec v1.0.3-0.20200929063507-e6143ca7d51d/go.m
|
||||
github.com/opencontainers/runtime-tools v0.0.0-20181011054405-1d69bd0f9c39/go.mod h1:r3f7wjNzSs2extwzU3Y+6pKfobzPh+kKFJ3ofN+3nfs=
|
||||
github.com/opencontainers/selinux v1.6.0/go.mod h1:VVGKuOLlE7v4PJyT6h7mNWvq1rzqiriPsEqVhc+svHE=
|
||||
github.com/opencontainers/selinux v1.8.0/go.mod h1:RScLhm78qiWa2gbVCcGkC7tCGdgk3ogry1nUQF8Evvo=
|
||||
github.com/otiai10/copy v1.6.0 h1:IinKAryFFuPONZ7cm6T6E2QX/vcJwSnlaA5lfoaXIiQ=
|
||||
github.com/otiai10/copy v1.6.0/go.mod h1:XWfuS3CrI0R6IE0FbgHsEazaXO8G0LpMp9o8tos0x4E=
|
||||
github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE=
|
||||
github.com/otiai10/curr v1.0.0 h1:TJIWdbX0B+kpNagQrjgq8bCMrbhiuX73M2XwgtDMoOI=
|
||||
github.com/otiai10/curr v1.0.0/go.mod h1:LskTG5wDwr8Rs+nNQ+1LlxRjAtTZZjtJW4rMXl6j4vs=
|
||||
github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo=
|
||||
github.com/otiai10/mint v1.3.2 h1:VYWnrP5fXmz1MXvjuUvcBrXSjGE6xjON+axB/UrpO3E=
|
||||
github.com/otiai10/mint v1.3.2/go.mod h1:/yxELlJQ0ufhjUwhshSj+wFjZ78CnZ48/1wtmBH1OTc=
|
||||
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
|
||||
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
|
||||
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
|
||||
@@ -647,6 +655,7 @@ golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
|
||||
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
|
||||
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
|
||||
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
|
||||
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
|
||||
11
install.sh
Normal file
11
install.sh
Normal file
@@ -0,0 +1,11 @@
|
||||
# This script setups up the project P2PRC
|
||||
# Call: sh install.sh <name of binary>
|
||||
echo '# Add the following paths to .bashrc or .zshrc based on the configuration you have set'
|
||||
echo export P2PRC=$PWD
|
||||
echo export PATH=$PWD:\${PATH}
|
||||
export P2PRC=${PWD}
|
||||
export PATH=${PWD}:${PATH}
|
||||
|
||||
# Expects an argument of the name of the binary
|
||||
go build -o ${1}
|
||||
./p2prc --dc
|
||||
@@ -1,201 +0,0 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -1,147 +0,0 @@
|
||||
# Ansible p2p-rendering-computation
|
||||
|
||||
This repository contains an Ansible playbook and instructions to create and manage a single (or many) bare metal deep learning machines. For a description of why Ansible was chosen and what other alternatives were considered, please see [ToolSelection.md](ToolSelection.md)
|
||||
|
||||
## Quick Reference
|
||||
|
||||
If you've already [installed Ansible](#Installation), you can execute the entire playbook by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml
|
||||
```
|
||||
|
||||
You can also execute only the pieces you need by passing tags on the command line:
|
||||
|
||||
- Install only apt/pip [pre-requisites](roles/packages) to execute the other roles:
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "packages"
|
||||
```
|
||||
- Install [Docker CE](roles/docker):
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "docker"
|
||||
```
|
||||
- Install the [Nvidia CUDA GPU drivers](roles/cuda):
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "cuda"
|
||||
```
|
||||
- Install the [Nvidia Docker Runtime](roles/nvidia):
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "nvidia"
|
||||
```
|
||||
|
||||
## What's Included
|
||||
|
||||
After running the ansible script your machines will be loaded with the following:
|
||||
|
||||
1. Docker
|
||||
2. Nvidia CUDA GPU Drivers
|
||||
3. Nvidia Docker Runtime
|
||||
4. TensorFlow GPU Python3 Docker Container
|
||||
5. JupyterLab
|
||||
|
||||
## Using This Repository to Configure Your Environment
|
||||
|
||||
1. [Installation](#Installation)
|
||||
2. [Configuration](#Configuration)
|
||||
3. [Running](#Running)
|
||||
|
||||
---
|
||||
|
||||
### Installation
|
||||
|
||||
Ansible runs on your local machine and sends commands to the remote (machine learning) machines. You'll need ansible installed locally (not on the machine learning boxes).
|
||||
For macOS users, the easiest way to install Ansible is via [Homebrew](https://brew.sh/):
|
||||
|
||||
```bash
|
||||
$ brew install ansible
|
||||
```
|
||||
|
||||
If that's not your cup of tea, install Ansible by following the directions for your machine [here](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-the-control-machine).
|
||||
|
||||
---
|
||||
|
||||
### Configuration
|
||||
|
||||
Gather the following:
|
||||
|
||||
- SSH key or user credentials for the remote account
|
||||
|
||||
**Note:** Ansible does not expose a channel to allow communication between the user and the ssh process to accept a password manually to decrypt an ssh key when using the ssh connection plugin (which is the default). The use of `ssh-agent` is highly recommended.
|
||||
|
||||
- List of servers you wish to manage:
|
||||
- hostnames/IP addresses
|
||||
- SSH port
|
||||
- usernames
|
||||
|
||||
Copy [hosts.example] to `/etc/ansible/hosts` (if it does not already exist). Populate the `hosts` file (no extension) with the information about the servers you gathered above.
|
||||
|
||||
Confirm that you have populated your Ansible hosts file correctly:
|
||||
|
||||
```bash
|
||||
$ ansible-inventory --list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Running
|
||||
|
||||
Once you're satisfied that you correctly populated your `hosts` file, update the `- hosts:` line of [tensorflow.yml] to reflect the hosts or groups you want to configure.
|
||||
|
||||
Examples:
|
||||
|
||||
- Apply against a single host defined as `ml2` in `/etc/ansible/hosts`:
|
||||
```yaml
|
||||
- hosts: ml2
|
||||
```
|
||||
- Apply against a group of hosts defined as `production` in `/etc/ansible/hosts`:
|
||||
```yaml
|
||||
- hosts: production
|
||||
```
|
||||
- Apply against all hosts defined in `/etc/ansible/hosts`:
|
||||
```yaml
|
||||
- hosts: all
|
||||
```
|
||||
|
||||
Then, when you're ready, run the playbook:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --ask-become-pass
|
||||
```
|
||||
|
||||
**Note:** You must have `sudo` access to run the playbook!
|
||||
|
||||
Review the output:
|
||||
|
||||
- `[ok]` means no change (this task was already completed)
|
||||
- `[changed]` means the task successfully ran and the change was applied
|
||||
- `[unreachable]` means the host could not be reached
|
||||
- `[failed]` means the task ran but failed to complete
|
||||
|
||||
`[ok]` and `[changed]` are successful outcomes. Any `[unreachable]` and `[failed]` outputs should be investigated and resolved.
|
||||
|
||||
**Note:** This Ansible playbook is idempotent; once a configuration has been successfully applied, if you apply it again, all actions will report `[ok]`.
|
||||
|
||||
## Executing Tensorflow Jobs in Your New Environment
|
||||
|
||||
1. Point your browser to http://<hostname>:8888 and login with the password you provided.
|
||||
2. The `jupyter.volumes.source` folder will be mounted as the `notebooks` folder.
|
||||
3. Edit and execute your Jupyter notebooks as normal!
|
||||
|
||||
### Command Line Access
|
||||
|
||||
If you need to drop into a GPU-powered TensorFlow environment, SSH into the remote machine and execute the following:
|
||||
|
||||
```bash
|
||||
$ docker run --runtime=nvidia -it --rm tensorflow/tensorflow:latest-gpu-py3 bash
|
||||
```
|
||||
|
||||
**Note:** You must be a member of the `docker` group or have `sudo` access on the _remote machine_ to execute docker commands.
|
||||
|
||||
---
|
||||
|
||||
## Additional Files
|
||||
|
||||
- [ansible.cfg](ansible.cfg) enables SSH credential forwarding. This is a necessary step during data synchronization, as Ansible delegates those credentials to the master/writer host to push the data folder out to each of the mirrors.
|
||||
- [Dockerfile](Dockerfile) is used to build the Arricor TensorFlow image. See [Docker.md](Docker.md) for additional details.
|
||||
- [hosts.example](hosts.example) is an example of the Ansible inventory hosts file saved in `/etc/ansible/hosts`
|
||||
- [secrets.example.yml](secrets.example.yml) is an example of the expected structure of the `secrets.yml` file
|
||||
@@ -1,93 +0,0 @@
|
||||
# Configuration Management Tool Selection
|
||||
|
||||
## Summary
|
||||
|
||||
Ansible is proposed as the configuration management tool for this project due to the following characteristics:
|
||||
|
||||
- no agent installation on the managed nodes
|
||||
- push-based configuration changes do not require a separate machine to be provisioned
|
||||
- commands can be pushed from any developer/operator workstation
|
||||
- all updates are performed over SSH
|
||||
|
||||
## Tools Considered
|
||||
|
||||
Where a tool offers an open source and an enterprise version (e.g., Puppet, Terraform), only the open source version was considered. Web or other graphical user interfaces (GUIs) were not considered, as the goal of this phase is to produce infrastructure as code which can be run (and re-run) from the command line interface (CLI).
|
||||
|
||||
- [Ansible](#Ansible)
|
||||
- [Chef](#Chef)
|
||||
- [Puppet](#Puppet)
|
||||
- [Salt](#Salt)
|
||||
- [Terraform](#Terraform)
|
||||
|
||||
### Ansible
|
||||
|
||||
[Ansible's simple interface and usability fit right into the sys admin mindset, and in a shop with lots of Linux and Unix systems, Ansible is quick and easy to run right out of the gate](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-Salt.html?page=4).
|
||||
|
||||
#### Ansible Pros
|
||||
|
||||
- Requires no agent installation on the managed nodes.
|
||||
- Push-based.
|
||||
- Does not require a separate machine to be provisioned. Commands can be pushed from any developer/operator workstation.
|
||||
- All updates are performed over SSH.
|
||||
|
||||
#### Ansible Cons
|
||||
|
||||
- Push-based synchronization means that the configuration is only refreshed on demand.
|
||||
|
||||
### Chef
|
||||
|
||||
[Chef has a stable and well-designed layout, and while it's not quite up to the level of Puppet in terms of raw features, it's a very capable solution. Chef may pose the most difficult learning curve to administrators who lack significant programming experience, but it could be the most natural fit for development-minded admins and development shops](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-Salt.html?page=4).
|
||||
|
||||
#### Chef Pros
|
||||
|
||||
- AWS OpsWorks is based on Chef; any future migration to AWS could benefit from the existing configuration files.
|
||||
|
||||
#### Chef Cons
|
||||
|
||||
- **(Fatal)** Requires a master server and a workstation that will need to be maintained separately from the already provisioned deep learning machines.
|
||||
- **(Fatal)** Requires an agent (client) to be installed on each machine.
|
||||
- Requires an on-premises Chef server and [Push Jobs](https://docs.chef.io/push_jobs.html) plugin to push configurations manually. Otherwise, managed machines must check in with the master on a schedule.
|
||||
|
||||
### Puppet
|
||||
|
||||
[Puppet is the most mature and probably the most approachable of the four from a usability standpoint, though a solid knowledge of Ruby is highly recommended. Puppet is not as streamlined as Ansible or Salt, and its configuration can get Byzantine at times. Puppet is the safest bet for heterogeneous environments, but you may find Ansible or Salt to be a better fit in a larger or more homogenous infrastructure](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html?page=4).
|
||||
|
||||
#### Puppet Pros
|
||||
|
||||
- Configuration can be kept up to date via a schedule or manual configuration pushes.
|
||||
|
||||
#### Puppet Cons
|
||||
|
||||
- **(Fatal)** Requires a "puppet master" (server) that will need to be maintained separately from the already provisioned machines.
|
||||
- **(Fatal)** Requires an agent (client) to be installed on each machine.
|
||||
|
||||
### Salt
|
||||
|
||||
[Salt is the sleekest and most robust of the four, and like Ansible it will resonate with sys admins. Highly scalable and quite capable, Salt is hamstrung only by the Web UI](https://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html?page=4).
|
||||
|
||||
#### Salt Pros
|
||||
|
||||
- Push-based.
|
||||
- Can function via regular SSH or via client agents called "minions".
|
||||
|
||||
#### Salt Cons
|
||||
|
||||
- When running agentless, push-based synchronization means that the configuration is only refreshed on demand.
|
||||
|
||||
### Terraform
|
||||
|
||||
The initial hope was that [Terraform](https://terraform.io) could be used for configuration management for this project. However, after reviewing the [available Terraform provider plugins](https://terraform.io/docs/providers) and other available material, it was agreed on 23 January 2019 that Terraform was not suitable for this stage of the project.
|
||||
|
||||
In the future, if Arricor migrates its instances to a cloud service provider (CSP) such as AWS, Azure, or Google Cloud Platform, this configuration may be reusable as a first step via the [terraform-provisioner-ansible](https://github.com/jonmorehouse/terraform-provisioner-ansible) plugin. This plugin allows the user to create CSP instances, then execute the given Ansible playbook against those instances.
|
||||
|
||||
#### Terraform Pros
|
||||
|
||||
- Cloud native with support for multiple service providers.
|
||||
|
||||
#### Terraform Cons
|
||||
|
||||
- **(Fatal)** No support for provisioning bare metal instances.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Ansible allows for management of a server from scratch, with only the requirement to have a valid username and password. Additionally, its agentless approach avoids having to configure and maintain an admin server, further simplifying operations at the scale Arricor requires.
|
||||
@@ -1,6 +0,0 @@
|
||||
[defaults]
|
||||
host_key_checking = False
|
||||
inventory = hosts
|
||||
|
||||
[ssh_connection]
|
||||
ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
all:
|
||||
vars:
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
test:
|
||||
hosts:
|
||||
test1:
|
||||
ansible_host: 0.0.0.0 # Replace with your remote IP
|
||||
ansible_port: 44003 # Replace with your remote SSH port
|
||||
ansible_user: master # Replace wtih your username
|
||||
ansible_ssh_pass: password
|
||||
ansible_sudo_pass: password
|
||||
@@ -1,25 +0,0 @@
|
||||
---
|
||||
all:
|
||||
vars:
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
test:
|
||||
hosts:
|
||||
test1:
|
||||
ansible_host: 127.0.0.1 # Replace with your remote IP
|
||||
ansible_port: 2222 # Replace with your remote SSH port
|
||||
ansible_user: vagrant # Replace wtih your username
|
||||
|
||||
production:
|
||||
hosts:
|
||||
writer:
|
||||
ansible_host: 192.168.1.100 # Replace with your remote IP
|
||||
ansible_port: 2202 # Replace with your remote SSH port
|
||||
ansible_user: ansible # Replace wtih your username
|
||||
mirror1:
|
||||
ansible_host: 192.168.1.101 # Replace with your remote IP
|
||||
ansible_port: 2202 # Replace with your remote SSH port
|
||||
ansible_user: ansible # Replace wtih your username
|
||||
mirror2:
|
||||
ansible_host: 192.168.1.102 # Replace with your remote IP
|
||||
ansible_port: 2202 # Replace with your remote SSH port
|
||||
ansible_user: ansible # Replace wtih your username
|
||||
@@ -1,38 +0,0 @@
|
||||
################################################################################
|
||||
# This file consists of three main sections:
|
||||
# 1. vars - variables used throughout the playbook are defined here. If you want to change packages or version, this is the place to do it.
|
||||
# 2. tasks - these are the steps that will be executed by the playbook. This is a straightforward, sequential playbook.
|
||||
# 3. handlers - these steps run only in response to specific events. If the event does not occur, the handler does not run.
|
||||
################################################################################
|
||||
---
|
||||
- hosts: test # Alternatively, 'canary', 'blue'|'green', 'all' according to your deployment strategy and your /etc/ansible/hosts file
|
||||
become: yes # All commands must run as root
|
||||
strategy: free # Allows all hosts to run to the end of the playbook as fast as they can
|
||||
|
||||
vars_files:
|
||||
- secrets.yml
|
||||
|
||||
roles:
|
||||
- role: packages # General Dependencies
|
||||
tags: [packages]
|
||||
|
||||
# - role: rsync # Rsync Setup
|
||||
# tags: [rsync]
|
||||
# vars:
|
||||
# sync: "{{ secrets.sync }}"
|
||||
|
||||
# - role: docker # General Docker setup
|
||||
# tags: [docker]
|
||||
# vars:
|
||||
# users: "{{ secrets.users }}"
|
||||
|
||||
- role: cuda # CUDA Driver Setup
|
||||
tags: [cuda]
|
||||
|
||||
# - role: nvidia # NVIDIA-Docker Runtime Setup
|
||||
# tags: [nvidia]
|
||||
|
||||
# - role: jupyter # Launch Jupyter Container in Docker
|
||||
# tags: [jupyter]
|
||||
# vars:
|
||||
# volumes: "{{ secrets.volumes }}"
|
||||
@@ -1,23 +0,0 @@
|
||||
# The `cuda` Role
|
||||
|
||||
The `cuda` role installs the Nvidia CUDA GPU repository and drivers.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "cuda"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Apt Repositories (Debian/Ubuntu)
|
||||
|
||||
- Nvidia CUDA
|
||||
|
||||
### Apt GPG Signing Keys (Debian/Ubuntu)
|
||||
|
||||
- cudatools <cudatools@nvidia.com>
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- cuda v10.x
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,5 +0,0 @@
|
||||
---
|
||||
# CUDA Driver Setup
|
||||
- name: reboot the machine
|
||||
reboot:
|
||||
listen: "reboot the machine"
|
||||
@@ -1,22 +0,0 @@
|
||||
---
|
||||
# Cuda Driver Package Installation
|
||||
- name: copy the CUDA repo onto the machine
|
||||
copy:
|
||||
src: "{{ cuda.apt.repo }}"
|
||||
dest: "{{ general.working_dir }}/{{ cuda.apt.repo }}"
|
||||
|
||||
- name: get the CUDA signing key
|
||||
apt_key:
|
||||
state: present
|
||||
url: "{{ cuda.apt.signing_key_url }}"
|
||||
|
||||
- name: add the CUDA repo
|
||||
apt:
|
||||
deb: "{{ general.working_dir }}/{{ cuda.apt.repo }}"
|
||||
state: present
|
||||
|
||||
- name: install the CUDA drivers
|
||||
apt:
|
||||
name: "{{ cuda.apt.package }}"
|
||||
update_cache: yes
|
||||
notify: "reboot the machine"
|
||||
@@ -1,3 +0,0 @@
|
||||
---
|
||||
- import_tasks: debian.yml
|
||||
when: ansible_facts['os_family']|lower == 'debian' # debian, ubuntu
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
cuda:
|
||||
apt:
|
||||
package: cuda=11.* # known good with 10.0.130-1
|
||||
repo: cuda-11-3_11.3.1-1_amd64.deb # Network install
|
||||
dependency: cuda-runtime-11-3_11.3.1-1_amd64.deb
|
||||
signing_key_url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
|
||||
|
||||
general:
|
||||
working_dir: /tmp
|
||||
@@ -1,33 +0,0 @@
|
||||
# The `docker` Role
|
||||
|
||||
The `docker` role installs the Docker CE repository and runtime packages.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "docker"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Apt Repositories (Debian/Ubuntu)
|
||||
|
||||
- Docker CE
|
||||
|
||||
### Apt GPG Signing Keys (Debian/Ubuntu)
|
||||
|
||||
- Docker Release (CE deb) <docker@docker.com>
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- docker-ce v5:18.x
|
||||
|
||||
### System Services
|
||||
|
||||
- `docker` (enabled and started)
|
||||
|
||||
### User Groups
|
||||
|
||||
- docker
|
||||
|
||||
All users listed under `secrets.users` in `secrets.yml` will be added to the `docker` group.
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
# Docker Package Installation
|
||||
- name: add Docker's official GPG key
|
||||
apt_key:
|
||||
state: present
|
||||
url: "{{ docker.apt.signing_key_url }}"
|
||||
|
||||
- name: add Docker apt repository
|
||||
apt_repository:
|
||||
repo: "{{ docker.apt.repo }}"
|
||||
state: present
|
||||
|
||||
- name: install Docker CE
|
||||
apt:
|
||||
name: "{{ docker.apt.package }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
@@ -1,23 +0,0 @@
|
||||
---
|
||||
# General Docker Setup
|
||||
- import_tasks: debian.yml
|
||||
when: ansible_facts['os_family']|lower == 'debian' # debian, ubuntu
|
||||
|
||||
- name: ensure the docker service starts at boot
|
||||
systemd:
|
||||
name: "{{ docker.service_name }}"
|
||||
enabled: yes
|
||||
state: started
|
||||
|
||||
- name: create Docker group to avoid running as root
|
||||
group:
|
||||
name: "{{ docker.group_name }}"
|
||||
state: present
|
||||
|
||||
- name: add users to Docker group
|
||||
user:
|
||||
append: yes
|
||||
groups:
|
||||
- "{{ docker.group_name }}"
|
||||
name: "{{ item }}"
|
||||
loop: "{{ users }}"
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
docker:
|
||||
apt:
|
||||
package: docker-ce=5:18.* # known good with 5:18.09.1~3-0~ubuntu-bionic
|
||||
repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
|
||||
signing_key_url: https://download.docker.com/linux/ubuntu/gpg
|
||||
group_name: docker
|
||||
service_name: docker
|
||||
@@ -1,31 +0,0 @@
|
||||
# The `nvidia` Role
|
||||
|
||||
The `nvidia` role installs the Nvidia Docker runtime, enabling access to Nvidia CUDA GPU hardware from within Docker containers.
|
||||
|
||||
This role may be executed independently by running:
|
||||
|
||||
```bash
|
||||
$ ansible-playbook packages.yml --tags "nvidia"
|
||||
```
|
||||
|
||||
On execution, this role installs the following directly to the remote machine's operating system:
|
||||
|
||||
### Apt Repositories (Debian/Ubuntu)
|
||||
|
||||
- libnvidia_container
|
||||
- nvidia_container_runtime
|
||||
- nvidia_docker
|
||||
|
||||
### Apt GPG Signing Keys (Debian/Ubuntu)
|
||||
|
||||
- NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>
|
||||
|
||||
### Operating System Packages (Debian/Ubuntu)
|
||||
|
||||
- nvidia-docker2 v2.x
|
||||
|
||||
Containers are launched with the Nvidia Docker runtime by passing the `--runtime=nvidia` flag, e.g.,
|
||||
|
||||
```bash
|
||||
$ docker run --rm --runtime=nvidia nvidia/cuda nvidia-smi
|
||||
```
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user