Difference between revisions of "Eosc/nuggets"
(Created page with "= Background = Smaller, tangible activities to be implemented under the EOSC-Nordic umbrella, in support of the NLPL virtual laboratory. = User Management = The project ma...") |
(→Accounting and Allocations) |
||
| (8 intermediate revisions by the same user not shown) | |||
| Line 8: | Line 8: | ||
The project maintains an email list for all users with access to the NLPL virtual laboratory. | The project maintains an email list for all users with access to the NLPL virtual laboratory. | ||
| − | + | We need to extract programmatically, on Saga and Puhti, which users are associated with | |
| − | NLPL billing project(s) | + | any of the NLPL billing project(s), which should be read from a local file. |
| + | This mechanism should be run as cron(8) job every night and rewrite another local | ||
| + | file with active email addresses (which is then used at NLPL headquarters to automatically update | ||
the set of mailing list subscribers). | the set of mailing list subscribers). | ||
| + | = File System Management = | ||
| + | |||
| + | With multiple users maintaining a shared collection of directories and files | ||
| + | (aka the community directory, the shared software and data installations that make up the | ||
| + | core of the virtual laboratory), we frequently run into permission challenges that | ||
| + | at times are tedious to resolve or require administrator priviliges (which the NLPL | ||
| + | developers cannot have). | ||
| + | We would like to see an interface where a cron(8) job regularly reads a specification | ||
| + | of permissions and such from a file (maintained by NLPL), quality-controls the contents, | ||
| + | and then executes corresponding actions, e.g. something like | ||
| + | |||
| + | <pre> | ||
| + | software/modules/etc/nlpl/tmp rm -r | ||
| + | software/modules/ chgrp -R nlpl | ||
| + | software/modules/ chmod g+rwX,o+rX | ||
| + | </pre> | ||
| + | |||
| + | = Module Usage Statistics = | ||
| + | |||
| + | On Abel, we used to have a crude mechanism tracking the use of NLPL software modules, viz. | ||
| + | generating a log entry every time a module definition was <code>load</code>ed. | ||
| + | This was accomplished by calling out to the logger(1) utility. | ||
| + | To generalize this approach, we would minimally need a mechanism for logging | ||
| + | remotely to a central NLPL server. | ||
| + | |||
| + | = Accounting and Allocations = | ||
| − | + | Each country has its own mechanisms for user management, allocations, | |
| + | and accounting. | ||
| + | In Norway, currently all NLPL users are organized in a single project | ||
| + | (for allocation), making it difficult for the project manager to keep | ||
| + | track of users, including their affiliations and period of association | ||
| + | with the project. | ||
| + | Project association and access to accounting information in both | ||
| + | Finland and Norway is through in-house web services. | ||
| + | Could one imagine building an API-like abstraction over the national | ||
| + | services, e.g. a RESTful interface to retrieve usage statistics for a | ||
| + | project or individual user in a specific period? | ||
| + | At present, there appears to be no systematic distinction for accounting | ||
| + | purposes between cpu vs. gpu usage (at least not in Norway). | ||
= Software Management = | = Software Management = | ||
| + | We are still struggling to provide fully uniform software environments | ||
| + | for NLPL users on Puhti and Saga (the current two instances of the | ||
| + | virtual laboratory). | ||
| + | Assuming we continue to build on the software modules infrastructure, | ||
| + | two challenges are: (a) the inventory of basic building blocks differs | ||
| + | (e.g. compilers, Python, CUDA, MPI) and (b) installation of add-on | ||
| + | modules is only partially automated. | ||
| − | + | Could one agree with CSC and Sigma2 on a standardized inventory of | |
| − | + | basic modules and a common mechanism to build on top of that, e.g. | |
| + | either EasyBuild or Spack? | ||
| + | since the spring of 2020, USIT and UiO-Inf have started to | ||
| + | [http://wiki.nlpl.eu/index.php/Eosc/easybuild evaluate EasyBuild] | ||
| + | for that purpose. | ||
Latest revision as of 06:51, 9 September 2020
Contents
Background
Smaller, tangible activities to be implemented under the EOSC-Nordic umbrella, in support of the NLPL virtual laboratory.
User Management
The project maintains an email list for all users with access to the NLPL virtual laboratory. We need to extract programmatically, on Saga and Puhti, which users are associated with any of the NLPL billing project(s), which should be read from a local file. This mechanism should be run as cron(8) job every night and rewrite another local file with active email addresses (which is then used at NLPL headquarters to automatically update the set of mailing list subscribers).
File System Management
With multiple users maintaining a shared collection of directories and files (aka the community directory, the shared software and data installations that make up the core of the virtual laboratory), we frequently run into permission challenges that at times are tedious to resolve or require administrator priviliges (which the NLPL developers cannot have). We would like to see an interface where a cron(8) job regularly reads a specification of permissions and such from a file (maintained by NLPL), quality-controls the contents, and then executes corresponding actions, e.g. something like
software/modules/etc/nlpl/tmp rm -r software/modules/ chgrp -R nlpl software/modules/ chmod g+rwX,o+rX
Module Usage Statistics
On Abel, we used to have a crude mechanism tracking the use of NLPL software modules, viz.
generating a log entry every time a module definition was loaded.
This was accomplished by calling out to the logger(1) utility.
To generalize this approach, we would minimally need a mechanism for logging
remotely to a central NLPL server.
Accounting and Allocations
Each country has its own mechanisms for user management, allocations, and accounting. In Norway, currently all NLPL users are organized in a single project (for allocation), making it difficult for the project manager to keep track of users, including their affiliations and period of association with the project. Project association and access to accounting information in both Finland and Norway is through in-house web services. Could one imagine building an API-like abstraction over the national services, e.g. a RESTful interface to retrieve usage statistics for a project or individual user in a specific period?
At present, there appears to be no systematic distinction for accounting purposes between cpu vs. gpu usage (at least not in Norway).
Software Management
We are still struggling to provide fully uniform software environments for NLPL users on Puhti and Saga (the current two instances of the virtual laboratory). Assuming we continue to build on the software modules infrastructure, two challenges are: (a) the inventory of basic building blocks differs (e.g. compilers, Python, CUDA, MPI) and (b) installation of add-on modules is only partially automated.
Could one agree with CSC and Sigma2 on a standardized inventory of basic modules and a common mechanism to build on top of that, e.g. either EasyBuild or Spack? since the spring of 2020, USIT and UiO-Inf have started to evaluate EasyBuild for that purpose.
