Commit 6d025b9d authored by Romain Reuillon's avatar Romain Reuillon
Browse files

Merge branch 'doc' into 'master'

Doc

See merge request openmole/openmole!10
parents aa308410 dfbd39c4
......@@ -12,15 +12,15 @@ assemble:
- '(cd build-system && sbt publishLocal)'
- '(cd libraries && sbt publishLocal)'
- '(cd openmole && ./generateSite $PWD/site)'
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_PRIVATE_KEY" >~/.ssh/identity
- chmod 400 ~/.ssh/identity
- ssh-keyscan -p 21022 docker.openmole.org > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- echo "put -r ./openmole/site/* next" >./tonext
- sftp -i ~/.ssh/identity -b ./tonext -P 21022 user@docker.openmole.org
#- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
#- mkdir -p ~/.ssh
#- chmod 700 ~/.ssh
#- echo "$SSH_PRIVATE_KEY" >~/.ssh/identity
#- chmod 400 ~/.ssh/identity
#- ssh-keyscan -p 21022 docker.openmole.org > ~/.ssh/known_hosts
#- chmod 644 ~/.ssh/known_hosts
#- echo "put -r ./openmole/site/* next" >./tonext
#- sftp -i ~/.ssh/identity -b ./tonext -P 21022 user@docker.openmole.org
only:
refs:
- 9-dev
......
......@@ -33,7 +33,7 @@ Due to a data loss in 2016, only the versions from OpenMOLE 6 are available.
@h2{Why is my ssh authentication not working?}
@h2{Why is my SSH authentication not working?}
When one of the SSH authentications you've added to OpenMOLE is marked as failed, you can try these few steps to identify the problem.
......
......@@ -2,26 +2,30 @@
@import org.openmole.site._
@import org.openmole.site.stylesheet._
@def overview = "Overview"
@h2{@overview}
This GUI is an advanced editor for OpenMOLE experiment description through OpenMOLE scripts. It allows editing, managing and running them.
@h2{Overview}
This GUI is an advanced editor for OpenMOLE experiment description through OpenMOLE scripts.
It allows editing, managing and running them.
The way to write these experiments is fully explained in the @aa("main documentation of OpenMOLE", href := DocumentationPages.embed.file).
We focus here on the way to manage them in the application.
@break
@br@br
Basically, in OpenMOLE GUI, you can store, editing files, run and monitor experiments, store authentication credentials for distributed computation.
The application runs in a browser (Firefox, Chrome, Opera, etc). The First time you run it, you are asked for a
password to encrypt your settings (server port, authentication credentials, etc). Your settings data (so not your projects, which are never
wiped out) are preserved so long as your password do not change.
For now OpenMOLE GUI looks like a web application but still run as a heavy client one. This should change in version 8 or 9.
In the OpenMOLE GUI, you can upload and edit files, run and monitor experiments, and store authentication credentials for distributed computation.
The application runs in a browser (Firefox or Chrome).
The First time you run it, you are asked for a password to encrypt your settings (server port, authentication credentials, etc).
Your settings data (so not your projects, which are never wiped out) are preserved so long as your password do not change.
For now, the OpenMOLE GUI looks like a web application but still runs as a heavy client one.
This should change in version 8 or 9.
@break
@br@br
@img(src := Resource.img.guiGuide.overview.file, center(100))
@h2{Starting a project}
Clicking on the main red button @i{New project} in the menu bar offers 3 choices:
......@@ -30,8 +34,8 @@ Selecting this option creates a new script file called @i{newProject.oms} in the
@h3{From market place}
Selecting this option pops up a dialog containing all the running projects contained in our Market Place.
The Market Place is a collection of projects built by users and offer different aspects of experiments that can be run on the
OpenMOLE platform with a large variety of programming codes, exploration methods. The sources can be found @aa("here", href := shared.link.repo.market).
The Market Place is a collection of projects built by users and offer different aspects of experiments that can be run on the OpenMOLE platform with a large variety of programming codes, exploration methods.
The sources can be found @aa("here", href := shared.link.repo.market).
Just select one of the market entries and click on the @i{Dowload} button to import it in the current folder.
@h3{From existing model sources}
......@@ -41,18 +45,20 @@ The wizard automatically distinguishes your model's programming language among J
Then it detects the potential inputs and outputs. For each input/output, the wizard generates a variable with a relevant name if possible.
At the end of the import process, the should run your script without you having to do anything!
@break
@br@br
To import your model, click on the @i{Your Model}. A dialog box pops up. Set your model path in it.
To import your model, click on the @i{Your Model}.
A dialog box pops up. Set your model path in it.
The system should now display the programming language, a list of detected inputs / outputs and the command to launch your code.
In most cases, you are almost done. Just press the @i{Build} button at the bottom: the wizard dialog disappears and the OpenMOLE script is generated in the wordDirectory with your uploaded code!
In most cases, you are almost done.
Just press the @i{Build} button at the bottom: the wizard dialog disappears and the OpenMOLE script is generated in the wordDirectory with your uploaded code!
However, you may sometimes want to make some modifications if you observe the system did not correctly detect your code, its inputs/outputs or its launching command.
@break
@br@br
@img(src := Resource.img.guiGuide.modelImport.file, center(100))
@break
@br@br
For each input / output, three actions can be triggered using the icons located on the same line:
@ul
......@@ -67,18 +73,20 @@ The launching command uses the names of the previously defined input / output va
It is reactive: if the name of the input/output changes, the launching command is updated with the corresponding name.
For the native codes (C, C++, Python, R, ...), the following syntax is required (automatically used): i{${}}.
@break
@br@br
The Netlogo applications working with the .nls extension should be previously archived. The system will extract the archives
and deal with the extensions as resources of the task (displayed in the resources tab)
A seed variable (for the Netlogo random genarator) is automatically generated and passed to the Netlogo Generator through the Netlogo Task.
@h2{File Management}
The OpenMOLE application essentially handles files: your model files, your model inputs our outputs, and the OpenMOLE scripts,
where you describe your experiment.
@break
@br@br
We distinguish multiple kinds of resources:
@ul
......@@ -90,18 +98,20 @@ We distinguish multiple kinds of resources:
@li
other external resources used as input for a model are editable in the application (CSV files, text files, ...), while binary files like images cannot be modified.
@br
These files are managed in a file system located in the left sidebar. This side bar offers basic tools for managing files
@break
@br@br
@img(src := Resource.img.guiGuide.files.file, center(90))
@break
@br@br
The current directory is shown at the top in the folder navigation area. When the folder hierarchy is too deep to fit in the bar, it will be replaced by "...".
Clicking on one folder of the stack sets it as the current folder. In the image above, the current directory is for example @i{Pi Computation}.
@break
@br@br
The tool area at the top concerns the current folder and provides with
@ul
......@@ -122,7 +132,7 @@ The tool area at the top concerns the current folder and provides with
@li
@b{refreshing} the content of the current folder
@break
@br
Then, each file on each line has a settings button permitting for @b{cloning} the current file, @b{downloading} it, @b{editing} its name or @b{removing} it.
Other actions are available depending on the context (ie the file extension):
......@@ -134,18 +144,21 @@ Other actions are available depending on the context (ie the file extension):
@li
@b{to OMS} for code source potentially carried by a Task (.jar, nlogo, .py, etc). It starts the model wizard.
@h2{Play and monitor executions}
When a .oms file is edited, a @i{Play button} appears in the top right corner to start the execution of the workflow.
Once the workflow has been started, the execution panel appears, listing information for each execution on a separate row.
At any time this execution panel can be closed (without aborting the current runnings) and re-opened by clicking on the
@a("running icon", href := anchor(overview))
@a("running icon", href := "#Overview")
@break
@br@br
@img(src := Resource.img.guiGuide.running.file, center(100))
@break
@br@br
The different statuses of the executions are:
@ul
@li
......@@ -158,11 +171,12 @@ The different statuses of the executions are:
@b{canceled}: the execution has been canceled (by means of the button)
@h2{Authentications}
In OpenMOLE, the computation load can be delegated to remote @aa("environments", href := DocumentationPages.scale.file).
When clicking on the @a("authentication icon", href := anchor(overview)), panel appears with the list (initially empty)
of all the defined authentications. To add one authentication, click on @i{New} button.
When clicking on the @a("authentication icon", href := "#Overview"), panel appears with the list (initially empty) of all the defined authentications.
To add one authentication, click on @i{New} button.
The currently supported authentications are:
@ul
@li
......@@ -177,30 +191,34 @@ The currently supported authentications are:
@b{Grid certificate} (.p12) for @aa("Grid Computing", href := shared.link.egi)@br
It only requires your EGI certificate file and the associated password. Click on No certificate to select your certificate file. It will be renamed to egi.p12. Note that only one EGI certificate is required (you will not need any other one!)
@br
@img(src := Resource.img.guiGuide.authentication.file, center(100))
@break
@br@br
An authentication can be removed by clicking on the @i{trash bin icon}.
An existing authentication can also be edited by clicking on the name of an authentication in the list.
@break
@br@br
Each time an authentication is added, a check is made on the mentioned environment (for the EGI ones, a list of VOs to be checked can be set in the EGI authentication settings).
If it fails, a red label appears. When clicking on it, the error stack appears.
@h2{Plugins}
New features can be dynamically inserted in the OpenMOLE platform through plugins.
Advanced users build their own plugins to express concepts that might not be present (yet) in OpenMOLE. In OpenMOLE, plugins take the form of jar files.
It can also be a JVM based model provided as a plugin. The way to build plugins in OpenMOLE is full described @aa("here", href := DocumentationPages.pluginDevelopment.file)
@break
To add a plugin, open the @a("plugin management panel", href := anchor(overview)).
@br@br
To add a plugin, open the @a("plugin management panel", href := "#Overview").
You can upload a new plugin by clicking on the blue top right-hand corner and selecting the corresponding jar file.
Once uploaded, the plugin appears in the list.
@break
@img(src := Resource.img.guiGuide.plugin.file, center(100))
@br@br
@break
@img(src := Resource.img.guiGuide.plugin.file, center(100))
\ No newline at end of file
......@@ -5,7 +5,8 @@
@h2{What is a Hook?}
@h2{Default hooks}
@h3{What is a Hook?}
Tasks in OpenMOLE are mute pieces of software.
They are not conceived to write files, display values, or generally present any side effects at all.
......@@ -22,7 +23,10 @@ Different hooks are available for different actions that need to be performed.
@h3{How to plug a hook to a task}
The @code{hook} keyword is used to save or display results generated during the execution of a workflow.
The generic way to use it is to write either @code{hook(workDirectory / "path/of/a/file.csv")} to save the results in a CSV file, or @code{hook display} to display the results in the standard output.
There is only one mandatory argument to specify, the kind of @code{output} you want:
@ul
@li{@code{hook display} to display the results in the standard output, note that it is completely equivalent to writing @code{hook(display)} or @code{hook(output = display)}
@li{@code{hook(workDirectory / "path/to/a/file.csv")} to save the results in a CSV file}
@br
......@@ -42,7 +46,7 @@ Let's consider this simple workflow:
// Define an exploration task
DirectSampling(
evaluation = (hello hook display),
evaluation = (hello hook(workDirectory / "results/helloTask_${i}.csv"),
sampling = i in (0 to 9)
)
""", name = "plug a hook")
......@@ -50,8 +54,7 @@ Let's consider this simple workflow:
@br
The @code{hook} is plugged to the end of the @code{hello} task in the @code{DirectSampling}, which means that every time @code{hello} finishes, the hook is executed.
Here it means that the dataflow will be printed in the standard output, so all the output values for @code{i} will be displayed.
Here it means that for each @code{i} value, the dataflow will be printed in files named @b{helloTask_1.csv}, @b{helloTask_2.csv}, etc., located in the @b{results} repository (which will be automatically created if it does not exist yet).
@h3{Default hooks}
......
......@@ -2,23 +2,78 @@
@import org.openmole.site.tools._
@import org.openmole.site.tools.api._
@import org.openmole.site.content.Environment._
@import DocumentationPages._
@h2{Using clusters with OpenMOLE}
@h3{Batch systems}
Many distributed computing environments offer @aa("batch processing", href := shared.link.batchProcessing) capabilities.
OpenMOLE supports most of the batch systems.
@br
Batch systems generally work by exposing an entry point on which the user can log in and submit jobs.
OpenMOLE accesses this entry point using SSH.
Different environments can be assigned to delegate the workload resulting of different tasks or groups of tasks.
However, not all clusters expose the same features, so options may vary from one environment to another.
@br@br
Before being able to use a batch system, you should first provide your @aa("authentication", href := gui.file) information to OpenMOLE.
@h3{Grouping}
You should also note that the use of a batch environment is generally not suited for short tasks, @i{i.e.} less than 1 minute for a cluster.
In case your tasks are short, you can group several executions with the keyword @code{by} in your workflow.
For instance, the workflow below groups the execution of @b{model} by 100 in each job submitted to the environment:
@br@br
@hl.openmole(s"""
// Define the variables that are transmitted between the tasks
val i = Val[Double]
val res = Val[Double]
// Define the model, here it is a simple task executing "res = i * 2", but it can be your model
val model =
ScalaTask("val res = i * 2") set (
inputs += i,
outputs += (i, res)
)
// Define a local environment
val env = LocalEnvironment(10)
// Make the model run on the the local environment
DirectSampling(
evaluation = model on env by 100 hook display,
sampling = i in (0.0 to 1000.0 by 1.0)
)
""")
Many distributed computing environments offer @aa("batch processing", href := shared.link.batchProcessing) capabilities. OpenMOLE supports most of the batch systems. Batch systems generally work by exposing an entry point on which the user can log in and submit jobs. OpenMOLE accesses this entry point using SSH.
Different environments can be assigned to delegate the workload resulting of different tasks or groups of tasks. However, not all clusters expose the same features, so options may vary from one environment to another.
@p You should first provide your @aa("authentication", href := DocumentationPages.gui.file) information to OpenMOLE to be able to use your batch system.
@h2{PBS}
@aa("PBS", href := shared.link.batchSystem) is a venerable batch system for clusters. It is also referred to as Torque. You may use a PBS computing environment as follows:
@br @hl.openmole("""
@aa("PBS", href := shared.link.batchSystem) is a venerable batch system for clusters.
It is also referred to as Torque.
You may use a PBS computing environment as follows:
@br@br
@hl.openmole("""
val env =
PBSEnvironment(
"login",
"machine.domain"
)""")
@p @provideOptions:
@br
@provideOptions:
@ul
@li{@port,}
@li{@sharedDirectory,}
......@@ -34,16 +89,24 @@ val env =
@li{@apiEntryTitle{flavour} Specify the declination of PBS installed on your cluster. You can choose between @hl.openmoleNoTest{Torque} (for the open source PBS/Torque) or @hl.openmoleNoTest{PBSPro}. Defaults to @hl.openmoleNoTest{flavour = Torque}}
@li{@name}
@h2{SGE}
To delegate some computation load to a @aa("SGE", href := shared.link.gridEngine) based cluster you can use the @hl.openmoleNoTest{SGEEnvironment} as follows:
@br @hl.openmole("""
To delegate some computation load to a @aa("SGE", href := shared.link.gridEngine) based cluster you can use the @code{SGEEnvironment} as follows:
@br@br
@hl.openmole("""
val env =
SGEEnvironment(
"login",
"machine.domain"
)""")
@p @provideOptions:
@br
@provideOptions:
@ul
@li{@port,}
@li{@sharedDirectory,}
......@@ -56,10 +119,15 @@ val env =
@li{@threads,}
@li{@name.}
@h2{Slurm}
To delegate the workload to a @aa("Slurm", href := shared.link.slurm) based cluster you can use the @hl.openmoleNoTest{SLURMEnvironment} as follows:
@br @hl.openmole("""
To delegate the workload to a @aa("Slurm", href := shared.link.slurm) based cluster you can use the @code{SLURMEnvironment} as follows:
@br@br
@hl.openmole("""
val env =
SLURMEnvironment(
"login",
......@@ -68,7 +136,10 @@ val env =
gres = List( Gres("resource", 1) ),
constraints = List("constraint1", "constraint2")
)""")
@p @provideOptions:
@br
@provideOptions:
@ul
@li{@port,}
@li{@sharedDirectory,}
......@@ -86,16 +157,24 @@ val env =
@li{@apiEntryTitle{constraints} a list of Slurm defined constraints which selected nodes must match,}
@li{@name.}
@h2{Condor}
@aa("Condor", href := shared.link.condor) clusters can be leveraged using the following syntax:
@br @hl.openmole("""
@br@br
@hl.openmole("""
val env =
CondorEnvironment(
"login",
"machine.domain"
)""")
@p @provideOptions:
@@br
@provideOptions:
@ul
@li{@port,}
@li{@sharedDirectory,}
......@@ -106,16 +185,24 @@ val env =
@li{@threads,}
@li{@name.}
@h2{OAR}
Similarly, @aa("OAR", href := shared.link.oar) clusters are reached as follows:
@br @hl.openmole("""
@br@br
@hl.openmole("""
val env =
OAREnvironment(
"login",
"machine.domain"
)""")
@p @provideOptions:
@br
@provideOptions:
@ul
@li{@port,}
@li{@sharedDirectory,}
......
......@@ -2,51 +2,82 @@
@import org.openmole.site.tools._
@import org.openmole.site.tools.api._
@import org.openmole.site.content.Environment._
@import DocumentationPages._
@aa("EGI", href := shared.link.egi) is a grid infrastructure gathering computing resources from all over the world.
The @aa("European Grid Infrastructure", href := shared.link.egi) is a grid infrastructure gathering computing resources from all over the world.
It is a very powerful computing environment, but transpires as technically challenging to use.
OpenMOLE makes it very simple to benefit from the grid.
@h2
Setup your grid credentials
You first need to set your EGI certificate. The way to achieve this is describe in the @aa("GUI guide", href := DocumentationPages.gui.file)")
Note: The following instructions explain how to setup the EGI authentication in console mode (not recommended).
@h2{Setting up an EGI authentication}
To delegate a task to EGI you need to register your certificate in OpenMOLE. In the console execute:
You first need to import your EGI certificate in OpenMOLE as described in the @aa("GUI guide", href := gui.file + "#Authentications").
@br @hl.openmole("""
EGIAuthentication() = P12Certificate(encrypted, "/path/to/your/certificate.p12")""", header = """def encrypted = "" """)
@p You need to execute this operation only once and for all. OpenMOLE persists this information in your preferences folder.
@h3{Authentication in console mode}
@h2
Submitting jobs to EGI
@p
To use EGI through DIRAC you should setup an @hl.openmoleNoTest{EGIAuthentication} as explained above using a @b{P12 certificate bundle}. Other methods are not supported by DIRAC. In order to use EGI you must be registered in a @b{Virtual Organisation (VO)}.
@br The VO is the only compulsory parameter when creating an EGI environment. Here the VO @i{biomed} is specified, but you can specify the EGI VO of your choice:
@br @hl.openmole("""
val env = EGIEnvironment("biomed")""")
@b{!!! This is not recommended !!!}
@p Basic options available for EGI are:
@ul
@li{@apiEntryTitle{cpuTime} the maximum duration for the job in terms of CPU consumption, for instance 1 hour,}
@li{@openMOLEMemory,}
@li{@apiEntryTitle{debug} generate debugging information about the execution node (hostname, date, memory, max number of file descriptors, user proxy, ...). Defaults to @hl.openmoleNoTest{debug = false}}
@li{@name.}
@p The @hl.openmoleNoTest{EGIEnvironment} can be tuned using the previous options as in this example:
@br @hl.openmole("""
val env =
EGIEnvironment(
"biomed",
cpuTime = 4 hours,
openMOLEMemory = 200 megabytes
)""")
@p The @hl.openmoleNoTest{EGIEnvironment} also accepts a set of more advanced options:
@br@br
In the console, execute the following:
@br @br
@hl.openmole("""
EGIAuthentication() = P12Certificate(encrypted, "/path/to/your/certificate.p12")
""", header = """def encrypted = "" """)
@br
You only need to execute this operation once.
OpenMOLE will store this information in your preferences folder.
@h2{Submitting jobs to EGI}
@h3{Mandatory parameter}
In order to use EGI you must be registered in a @b{Virtual Organisation} (VO).
The VO is the only compulsory parameter when creating an EGI environment within OpenMOLE.
In the following example the VO @i{biomed} is specified, but you can use any VO:
@br@br
@hl.openmole("""
val env = EGIEnvironment("biomed")
""")
@h3{Optional parameters}
Other optional parameters are available when defining an @code{EGIEnvironment}:
@ul
@li{@apiEntryTitle{cpuTime} the maximum duration for the job in terms of CPU consumption, for instance 1 hour,}
@li{@openMOLEMemory,}
@li{@apiEntryTitle{debug} generate debugging information about the execution node (hostname, date, memory, max number of file descriptors, user proxy, ...). Defaults to @hl.openmoleNoTest{debug = false},}
@li{@name.}
Here is a use example using these parameters:
@br@br
@hl.openmole("""
val env =
EGIEnvironment(
"biomed",
cpuTime = 4 hours,
openMOLEMemory = 200 megabytes
)
""")
@h3{Advanced parameters}
The @code{EGIEnvironment} also accepts a set of more advanced options:
@ul
@li{@apiEntryTitle{service} a DIRAC REST API,}
@li{@apiEntryTitle{group} the name of the DIRAC group,}
......@@ -55,3 +86,33 @@ EGIAuthentication() = P12Certificate(encrypted, "/path/to/your/certificate.p12")
@li{@apiEntryTitle{fqan} additional flags for authentication,}