Infrastructure/codface - Revision history

Oe: /* (1) Sharing of Training (and Evaluation) Data */

2025-08-17T21:26:13Z

‎(1) Sharing of Training (and Evaluation) Data

Oe: /* (0) CodFace: LLM Infrastructure */

2025-08-17T21:25:01Z

‎(0) CodFace: LLM Infrastructure

Oe at 21:23, 17 August 2025

2025-08-17T21:23:17Z

Oe: /* (2) Large Language Model Repository */

2025-08-15T10:15:25Z

‎(2) Large Language Model Repository

Oe: /* Design and Implementation */

2025-06-26T21:51:31Z

‎Design and Implementation

Oe: /* Design and Implementation */

2025-06-26T21:44:53Z

‎Design and Implementation

Oe: /* Design and Implementation */

2025-06-26T21:43:55Z

‎Design and Implementation

Oe: /* (1) Sharing of Training (and Evaluation) Data */

2025-06-26T21:10:21Z

‎(1) Sharing of Training (and Evaluation) Data

Oe: /* (0) CodFace: LLM Infrastructure */

2025-06-23T06:55:51Z

‎(0) CodFace: LLM Infrastructure

Oe at 08:59, 22 June 2025

2025-06-22T08:59:16Z

@@ Line 12: / Line 12: @@
 = (1) Sharing of Training (and Evaluation) Data =
-LLM pre-training requires large volumes of textual data, often a few terabytes of compressed JSONlines files.
+LLM pre-training requires large volumes of textual data, often many terabytes of compressed JSONlines files.
 Common data sets – like e.g. [https://github.com/allenai/allennlp/discussions/5265 mC4], [https://huggingface.co/datasets/HuggingFaceFW/fineweb FineWeb] or [https://hplt-project.org/datasets/v2.0 HPLT] – provide metadata that can be used for selection, filtering, or inspection.
 Similarly (if to a lesser degree), model adaptation – fine-tuning, domain or task adaptation, alignment, etc. – and evaluation also build on shared public data sets, if typically of much smaller size.

@@ Line 7: / Line 7: @@
 For technical and strategic reasons, it is tempting to explore reduced dependency on the Hugging Face ecosystem, which after all is a closed and commercially motivated service.
 Following on the NLPL notion of a “virtual laboratory”, it will be worthwhile to work towards infrastructure and services for data and model sharing across HPC systems in a Nordic or European perspective, such that LLM development increasingly can build on a unified environment and so that duplicative effort is reduced.
-The UiO Language Technology Group (LTG) coordinates data management in the European [https://hplt-project.org/ HPLT] (where also Sigma2 is a consortium member) and [https://openeurollm.eu/ OpenEuroLLM] projects, where the work plan minimally calls for providing an LLM download repository.
+The UiO Language Technology Group (LTG) coordinates data management in the European project [https://hplt-project.org/ HPLT] (where also Sigma2 is a consortium member) and [https://openeurollm.eu/ OpenEuroLLM], where the work plan minimally calls for providing an LLM download repository.
-In a broader perspective, open and self-hosted LLM infrastructures will also be required in other initiatives, e.g. the Norwegian AI Factory and the Language Model Factory at the Norwegian National Library.
+In a broader perspective, open and self-hosted LLM infrastructures will also be required in other initiatives, e.g. the Norwegian AI Cloud (NAIC) Norwegian AI Factory (NAIF) and the Language Model Factory at the Norwegian National Library.
 = (1) Sharing of Training (and Evaluation) Data =

@@ Line 21: / Line 21: @@
 To users, ideally, this should appear like a path in the local file system.
 Such on-line sharing of LLM data across EuroHPC systems is a task in [https://github.com/OpenEuroLLM/training-data-catalogue/blob/main/README.md WP3 of OpenEuroLLM] (coordinated by UiO), initially in the form of a curated community directory mirrorred across systems.
 = (2) Large Language Model Repository =

@@ Line 21: / Line 21: @@
 To users, ideally, this should appear like a path in the local file system.
 Such on-line sharing of LLM data across EuroHPC systems is a task in [https://github.com/OpenEuroLLM/training-data-catalogue/blob/main/README.md WP3 of OpenEuroLLM] (coordinated by UiO), initially in the form of a curated community directory mirrorred across systems.
 = (2) Large Language Model Repository =

@@ Line 42: / Line 42: @@
 Following the Hugging Face experience, LTG suggests organizing the core of the model repository in Git, with Large File Storage (LFS) support.
 In this scheme, each model would be organized as one Git repository, which naturally supports core notions like ownership, management of access rights, versioning (aka revision management), and bundling of files and metadata that jointly comprise each model.
-For the HPLT model repository pilot, it will be desirable that Sigma2 provide and maintain a Git service, that initially would need to support at least a few thousand repositories (amounting to a few terabytes of storage) and write access by a handful of HPLT users.
+For the HPLT model repository pilot, it will be desirable that Sigma2 provide and maintain a scalable Git service, that initially would need to support at least a few thousand repositories (amounting to a few terabytes of storage) and write access by a handful of HPLT users – preferably early in the fall.
 For model distribution, a designated web service (for faceted browsing and download) would have to be developed.

@@ Line 47: / Line 47: @@
 LTG and HPLT partner Prompsit could take part in the design and implementation of the web interface, but it should preferably be hosted and operated by Sigma2.
 Technically, one can imagine a solution where a dedicated server (e.g. a virtual machine) continually keeps an on-disk clone of all Git model repositories and runs a web service, serving dynamic pages (based on a suitable framework).
-The [https://opus.nlpl.eu NLPL Opus
+This design could draw inspiration from the [https://opus.nlpl.eu NLPL Opus Parallel Corpus Repository], recently upgraded by HPLT members Prompsit and the University of Oslo, and hosted in the CSC cloud.
 How to facilitate shared responsibility (over time) and degrees of “community self-help”?

@@ Line 35: / Line 35: @@
 The [https://vectors.nlpl.eu NLPL Vectors Repository] was an early attempt at building and sharing a systematic collection of (old-school) language models, including some structured metadata.
-The existing repository contains about 220 models (most of them “classic” or contextualized word embeddings), available for download through a faceted search interface and for direct loading from the NLPL community directories on Saga and Puhti.
+The existing repository contains about 220 models (most of them “classic” or contextualized word embeddings), available for download through a (simple) faceted search interface and for direct loading from the NLPL community directories on Saga and Puhti.
 However, packaging each model as a Zip file is not directly interoperable with the popular Hugging Face ecosystem.
-The organization of models and metadata should be redesigned, hosting and synchronization across systems consolidated, and the “vectors repository” re-branded as something more modern, e.g. the ''NLPL Language Model Repository''.
+The organization of models and metadata should be redesigned, storage, download, and synchronization across systems consolidated, and the “vectors repository” re-branded as something more modern, e.g. the ''NLPL Model Repository''.
 This is a task in WP6 of HPLT (coordinated by UiO, with involvement of Sigma2) that needs to be completed in the fall of 2025.
 Following the Hugging Face experience, LTG suggests organizing the core of the model repository in Git, with Large File Storage (LFS) support.
 In this scheme, each model would be organized as one Git repository, which naturally supports core notions like ownership, management of access rights, versioning (aka revision management), and bundling of files and metadata that jointly comprise each model.
-For the HPLT model repository pilot, it would be desirable that Sigma2 provide and maintain a Git service, that initially would need to support at least a few thousand repositories (amounting to a few terabytes of storage) and write access by a handful of HPLT users.
+For the HPLT model repository pilot, it will be desirable that Sigma2 provide and maintain a Git service, that initially would need to support at least a few thousand repositories (amounting to a few terabytes of storage) and write access by a handful of HPLT users.
 For model distribution, a designated web service (for faceted browsing and download) would have to be developed.
 LTG and HPLT partner Prompsit could take part in the design and implementation of the web interface, but it should preferably be hosted and operated by Sigma2.
-Technically, one can imagine
+Technically, one can imagine a solution where a dedicated server (e.g. a virtual machine) continually keeps an on-disk clone of all Git model repositories and runs a web service, serving dynamic pages (based on a suitable framework).
-as a Management of metadata and versioned updates?
+How to facilitate shared responsibility (over time) and degrees of “community self-help”?
 Mechanisms for user contributions, community discussion, ticket management, and such?

@@ Line 1: / Line 1: @@
-= CodFace: LLM Infrastructure =
+= (0) CodFace: LLM Infrastructure =
 This page is an internal discussion document.
@@ Line 10: / Line 10: @@
 In a broader perspective, open and self-hosted LLM infrastructures will also be required in other initiatives, e.g. the Norwegian AI Factory and the Norwegian Language Model Factory.
-= Data Sharing =
+= (1) Sharing of Training (and Evaluation) Data =
 LLM pre-training requires large volumes of textual data, often a few terabytes of compressed JSONlines files.
@@ Line 17: / Line 17: @@
 Currently, different projects (or individual users) tend to establish and manage their own copies of relevant data on the system where they work.
-An alternative solution could be a centralized repository that is made available (read-only) on relevant HPC systems, for example using a tiered, caching file system.
+An alternative solution would be a centralized repository that is made available (read-only and world-readable) on relevant HPC systems, for example using a tiered, caching file system.
 To keep things simple, this could be implemented without any notion of access control rights, which would limit the scope of the repository to publicly available resources.
 To users, ideally, this should appear like a path in the local file system.
-On-line sharing of LLM data across EuroHPC systems is a task in WP3 of OpenEuroLLM (coordinated by UiO).
+Such on-line sharing of LLM data across EuroHPC systems is a task in [https://github.com/OpenEuroLLM/training-data-catalogue/blob/main/README.md WP3 of OpenEuroLLM] (coordinated by UiO), initially in the form of a curated community directory mirrorred across systems.
-= Model Repository =
+= (2) Large Language Model Repository =
 Abstractly parallel, but with a larger and more diverse target user group, pre-trained LLMs now are a part of much and varied AI work.
-MSc and PhD student projects often want to build on off-the-shelf models in a framework like Hugging Face Transformers.
+MSc and PhD student projects often want to build on off-the-shelf models in a framework like [https://huggingface.co/docs/transformers/index Hugging Face Transformers].
 The [https://vectors.nlpl.eu NLPL Vectors Repository] was an early attempt at building and sharing a systematic collection of (old-school) language models, including some structured metadata.
 The existing repository contains about 220 models (most of them “classic” or contextualized word embeddings), available for download through a faceted search interface and for direct loading from the NLPL community directories on Saga and Puhti.
 However, packaging each model as a Zip file is not directly interoperable with the popular Hugging Face ecosystem.
-The organization of models and metadata should be redesigned, hosting and synchronization across systems consolidated, and the “vectors repository” re-branded as something more modern, e.g. the ''Nordic Language Model Repository''.
+Following the Hugging Face experience, LTG suggests organizing the core of the model repository in Git, with Large File Storage (LFS) support.
-This is a task in WP6 of HPLT (coordinated by UiO).
+In this scheme, each model would be organized as one Git repository, which naturally supports core notions like ownership, management of access rights, versioning (aka revision management), and bundling of files and metadata that jointly comprise each model.
-= Implementation Notes =
+For model distribution, a designated web service (for faceted browsing and download) would have to be developed.
-Management of metadata and versioned updates?
+as a Management of metadata and versioned updates?
 Master repository in Git, object storage, a managed directory collection, or something else?
@@ Line 46: / Line 58: @@
 Sharing across clusters and to compute nodes without external access, e.g. [https://cvmfs.readthedocs.io/en/stable/ CernVMFS]?
-Additionally, one probably wants a download service and searchable index.
+= (3) Inference Endpoints =
-[https://chat.llm.sigma2.no LTG Pilot at NIRD]
+The current limited [https://chat.llm.sigma2.no LTG API Pilot] is built on the [https://github.com/gradio-app/gradio Gradio framework] and hosted at NIRD.