Dataset and Model Catalogue
This catalogue provides an overview of the datasets and open-source large machine learning (ML) and artificial intelligence (AI) models currently available across the MINERVA computing infrastructure. It serves as a central access point for researchers and developers to discover, evaluate, and utilize resources hosted on the participating HPC clusters.
The catalogue has been compiled through collaboration among project partners and is continuously updated as new datasets and models become available. By bringing together information from different infrastructures, it facilitates resource discovery, promotes reuse, and supports the development, training, fine-tuning, and evaluation of advanced AI applications.
The resources presented below are organized by HPC site and include details on the datasets and models currently hosted on each infrastructure. For every resource, the relevant storage paths are provided to support efficient access, deployment, and utilization by project partners across the MINERVA ecosystem.
Cluster Overview
| HPC Cluster | Hosting Institution | Country | Available Datasets | Available Models | System Information |
|---|---|---|---|---|---|
| Jean Zay | IDRIS | France | 0 datasets | 0 models | Link |
| Adastra | CINES | France | 0 datasets | 0 models | Link |
| Jupiter | FZJ | Germany | 0 datasets | 0 models | Link |
| Leonardo | CINECA | Italy | 23 datasets | 20 models | Link |
| MareNostrum 5 | BSC | Spain | 9 datasets | 33 models | Link |
Easy Access
MareNostrum 5:
# Load module to list models/datasets
> module load ai-hub
# Show the datasets that you have access to at the moment
> ai-hub –list datasets
# Show models that you have access to at the moment
> ai-hub –list models
Leonardo:
# Load module to list models/datasets
> module load ….
# module list
> ….
Available Datasets
The following section provides a consolidated view of the datasets currently deployed across the MINERVA HPC infrastructure. Each entry includes key metadata as well as the corresponding storage locations on the participating supercomputing systems, enabling straightforward access and reproducibility of workflows across sites.
The datasets listed are intended to support a wide range of machine learning and artificial intelligence use cases, including training, fine-tuning, benchmarking, and evaluation. Availability may vary across systems depending on local storage and replication policies.
| Category | Dataset | Description | License | Leonardo Path | MN5 Path |
|---|---|---|---|---|---|
| Pretraining Multilingual | Common Crawl | The Common Crawl organization collects petabytes of data on the web, regularly collected since 2008. The corpus contains raw web page data, metadata extracts, and text extracts. | license | /leonardo_work/MNRVA_datasets/datasets/commoncrawl | /gpfs/scratch/shared/ai-hub/datasets/minerva/common_crawl (restricted) |
| Pretraining Multilingual | HPLTv2 | Large-scale multilingual text corpus extracted from web archives (mainly Internet Archive and Common Crawl), cleaned, deduplicated, and language-identified | CCO | /leonardo_work/MNRVA_datasets/datasets/HPLTv2 | /gpfs/scratch/shared/ai-hub/datasets/minerva/HPLT (restricted) |
| Pretraining Multilingual | Fineweb-2 | A high-quality, multilingual pre-training dataset derived from nearly 100 Common Crawl snapshots (2013-2024). It’s the second iteration of FineWeb, created using an adaptive curation pipeline for improved language Identification, deduplication, and filtering, focusing on non-English content. | odc-by | /leonardo_work/MNRVA_datasets/datasets/Fineweb-2 | /gpfs/scratch/shared/ai-hub/datasets/minerva/HuggingFaceFW/fineweb-2 (restricted) |
| Pretraining Multilingual | c4 | A cleaned version of Common Crawl’s web crawl corpus. | odc-by | /leonardo_work/MNRVA_datasets/datasets/c4 | /gpfs/scratch/shared/ai-hub/datasets/minerva/allenai/c4/gpfs/scratch/shared/ai-hub/datasets/minerva/allenai/c4 (restricted) |
| Pretraining Multilingual | Oscar | It is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture. Data is distributed by language in both original and deduplicated form. | cc0-1.0 | /leonardo_work/MNRVA_datasets/datasets/Oscar | restricted |
| Pretraining Multilingual | CulturaX | Created by combining and extensively deduplicating the latest versions of mC4 and all available OSCAR corpora (up to 23.01 distribution) | mC4 license and OSCAR license | /leonardo_work/MNRVA_datasets/datasets/CulturaX | Not deployed |
| Pretraining Multilingual | common_corpus | Highly curated dataset beyond typical web-crawled data, sourcing content from diverse domains like cultural heritage, government and legal, science, and code | All data is uncopyrighted or permissibly licensed | /leonardo_work/MNRVA_datasets/datasets/common_corpus | /gpfs/scratch/shared/ai-hub/datasets/minerva/PleIAs/common_corpus (restricted) |
| Pretraining English | Nemotron-CC | It is derived from Common Crawl and is distinguished by its use of model-based quality filtering (like classifier ensembling) and the inclusion of a significant amount of synthetically generated data. | subject to the Common Crawl Terms of Use | /leonardo_work/MNRVA_datasets/datasets/Nemotron-CC | Not deployed |
| Pretraining | |||||
| English | dclm-baseline-1.0 | It is a high-quality, filtered, and deduplicated subset of the massive DCLM-Pool (which itself is 240T tokens of Common Crawl data). | CC-by-4.0 | /leonardo_work/MNRVA_datasets/datasets/dclm-baseline-1.0 | /gpfs/scratch/shared/ai-hub/datasets/minerva/dclm-baseline-1.0 (restricted) |
| Pretraining | |||||
| English | SlimPajama-627B | It is a cleaned and deduplicated version of Together’s RedPajama. | Apache 2.0 | /leonardo_work/MNRVA_datasets/datasets/SlimPajama-627B | Not deployed |
| Pretraining English | Pile | Composed of 22 high-quality sub-datasets, designed to improve the cross-domain generalization capability of LLMs | refer to the specific license depending on the subset you use, PubMed Central: MIT | /leonardo_work/MNRVA_datasets/datasets/Pile | Not deployed |
| Reasoning | OpenThoughts3-1.2M | A chain-of-thought reasoning, containing high-quality questions and step-by-step reasoning traces distilled from the QwQ-32B teacher model | Apache 2.0 | /leonardo_work/MNRVA_datasets/datasets/OpenThoughts3-1.2M | /gpfs/scratch/shared/ai-hub/datasets/minerva/open-thoughts/OpenThoughts3-1.2M (restricted) |
| Post-training Multilingual | Aya Collection | The largest open-source multilingual instruction fine-tuning dataset to date. It aggregates: a human-curated native dataset (Aya Dataset), templated versions of existing multilingual NLP datasets, and machine translations of English instruction fine-tuning datasets. | Apache 2.0 | /leonardo_work/MNRVA_datasets/datasets/aya_collection | /gpfs/scratch/shared/ai-hub/datasets/minerva/CohereLabs/aya_collection (restricted) |
| Post-training Multilingual | OASST2 | Human-generated, multi-turn dialogue dataset structured as conversation trees, ideal for instruction tuning and RLHF (Reward Models). | Apache 2.0 | /leonardo_work/MNRVA_datasets/datasets/OASST2 | /gpfs/scratch/shared/ai-hub/datasets/minerva/OpenAssistant/oasst2 (restricted) |
| Mix Post-training | Tulu 3 Datasets: tulu-3-sft-mixture, llama-3.1-tulu-3-8b-preference-mixture, llama-3.1-tulu-3-70b-preference-mixture, llama-3.1-tulu-3-405b-preference-mixture | SFT and DPO datasets for various tasks | odc-by | /leonardo_work/MNRVA_datasets/datasets/tulu3 | Not deployed |
| Post-training | HelpSteer1, HelpSteer2, HelpSteep3 | A multi-attribute (helpfulness, correctness, coherence), human-annotated preference dataset used to train aligned LLMs via techniques like DPO or SteerLM | cc-by-4.0 | /leonardo_work/MNRVA_datasets/datasets/HelpSteer1 | |
| /leonardo_work/MNRVA_datasets/datasets/HelpSteer2 | |||||
| /leonardo_work/MNRVA_datasets/datasets/HelpSteer3 | Not deployed | ||||
| Post-training | ultrafeedback | AI feedback preference dataset annotated by GPT-4 (critiques/scores), used for DPO/alignment | MIT | /leonardo_work/MNRVA_datasets/datasets/ultrafeedback | Not deployed |
| Parallel Data | Tatoeba Challenge | Massive multilingual machine translation dataset/benchmark mixing shuffled OPUS training data with high-quality, crowd-sourced Tatoeba test data | license | /leonardo_work/MNRVA_datasets/datasets/Tatoeba_Challenge | Not deployed |
| Code | StarCoder | Pre-training corpus for code LLMs, consists of source code, GitHub issues, Jupyter notebooks, and commit messages | BigCode OpenRAIL-M | /leonardo_work/MNRVA_datasets/datasets/StarCoder | Not deployed |
| Code | tulu-3-sft-personas-code | Post-training dataset, persona-driven synthetic SFT data targeting code generation and debugging | odc-by | Not deployed | Not deployed |
| Math | FineMath | A massive pre-training dataset focused on English math | odc-by | /leonardo_work/MNRVA_datasets/datasets/FineMath | Not deployed |
| Math | InfiMM-WebMath-40B | A Multimodal Pre-training dataset, designed for mathematical reasoning | odc-by | /leonardo_work/MNRVA_datasets/datasets/InfiMM-WebMath-40B | Not deployed |
| Math | tulu-3-sft-personas-math, tulu-3-sft-personas-math-grade, tulu-3-sft-personas-algebra | Post-training dataset, Synthetic SFT Data in the instruction-tuning format, with complex math problems and detailed, step-by-step solutions that often use LaTeX for mathematical notation | odc-by | Not deployed | Not deployed |
| Safety | coconot, wildjailbreak, wildguardmix | Post-training datasets focused on LLM safety and alignment: noncompliance (refusals) and jailbreaking | License, odc-by, odc-by | Not deployed | Not deployed |
| Multi-modal (Image-Text) | LAION-400m, LAION5B, Laion-coco, Laion translated, LAION POP, LAION5B High-Res, LAION-Aesthetics, Re-LAION-5B research, Re-LAION-5B research-safe | CLIP-filtered image-text pairs, including specialized subsets enhanced by VLM/synthetic captions or filtered for high resolution, aesthetics, and safety (Re-LAION) | CC-BY 4.0 | ||
| , CC-BY 4.0 | |||||
| ,_,_,_,_,_, Apache 2.0, Apache 2.0 | /leonardo_work/MNRVA_datasets/datasets/LAION | Not deployed | |||
| Multi-modal (Image-Text) | DataComp-1.4B | Highly filtered, high-quality English multimodal dataset for Language-Vision pre-training (e.g., CLIP), derived from CommonPool | cc-by-4.0 | /leonardo_work/MNRVA_datasets/datasets/DataComp-1.4B | Not deployed |
Notes:
“Not deployed” indicates the model/dataset is not available on the corresponding system.
“Restricted” appended to a path indicates that the model/dataset is present on the system but subject to access limitations imposed by the hosting centre.
Model/Dataset paths are provided for transparency and reproducibility within the MINERVA infrastructure.
Access policies may vary across HPC centres depending on licensing and governance constraints. In case of any problems encountered when accessing a dataset or model, users should contact the MINERVA AI support team, specifying the issue and the EuroHPC system being used.
Available Models
This section lists the open-source machine learning and artificial intelligence models deployed across the MINERVA HPC infrastructure. The catalogue includes models intended for training, fine-tuning, inference, and benchmarking across heterogeneous systems.
Model availability varies depending on licensing conditions and local deployment policies, but storage paths are provided for all systems where models are installed.
| Model | name Size Type Acceptance | needed License ADASTRA | Path Leonardo | Path MN5 | Path |
|---|---|---|---|---|---|
| SmolLM3-3B-Base 1-3B Language | (Multilingual) No 8 | languages | |||
| Apache-2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/SmolLM3-3B-Base /gpfs/scratch/shared/ai-hub/models/text-models/HuggingFaceTB/SmolLM3-3B-Base (restricted) | ||||
| Llama-3.2-1B 1-3B Language | (Multilingual) Yes 8 | languages | |||
| llama | 3.2 Not | deployed /leonardo_work/MNRVA_datasets/models/Llama-3.2-1B /gpfs/ | scratch/shared/ai-hub/models/text-models/meta-llama/Llama-3.2/ | Llama-3.2-1B-Instruct (restricted) | |
| Qwen3-1.7B-Base 1-3B Language | (Multilingual) No 119 | languages | |||
| Apache-2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Qwen3-1.7B-Base /gpfs/ | scratch/shared/ai-hub/models/text-models/Qwen/Qwen3/Qwen3-1.7B-Base (restricted) | |||
| gpt-sw3-1.3b 1-3B Language | (Multilingual) No Nordic | ||||
| Apache-2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/gpt-sw3-1.3b /gpfs/ | scratch/shared/ai-hub/models/text-models/AI-Sweden-Model/gpt-sw3-1.3b (restricted) | |||
| EuroLLM-1.7B 1-3B Language | (Multilingual) No 35 | languages | |||
| Apache-2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/EuroLLM-1.7B /gpfs/ | scratch/shared/ai-hub/models/text-models/utter-project/EuroLLM-1.7B (restricted) | |||
| open-sci-ref-v0.01-1.7b-nemotron-hq-1T-8192 1-3B Language | (Multilingual) No 1.7B | params | |||
| Trained | on | 1T | tokens | ||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/open-sci-ref-v0.01-1.7b-nemotron-hq-1T-8192 /gpfs/scratch/shared/ai-hub/models/text-models/open-sci/open-sci-ref-v0.01-1.7b-nemotron-hq-1T-8192 (restricted) | |||
| Qwen2-VL-2B 1-3B Vision/language No Multilingual | |||||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Qwen2-VL-2B /gpfs/scratch/shared/ai-hub/models/multimodal-models/Qwen/Qwen2/Qwen2-VL-2B (restricted) | |||
| TowerVision-2B 1-3B Vision/language No 18 | languages | ||||
| cc-by-nc-sa-4.0 Not | deployed /leonardo_work/MNRVA_datasets/models/TowerVision-2B /gpfs/scratch/shared/ai-hub/models/multimodal-models/utter-project/TowerVision-2B (restricted) | ||||
| siglip2-giant-opt-patch16-384 1-3B Vision/language No Multilingual | |||||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/siglip2-giant-opt-patch16-384 /gpfs/scratch/shared/ai-hub/models/image-classification/google/siglip2-giant-opt-patch16-384 (restricted) | |||
| CLIP-ViT-L-14-laion2B-s32B-b82K 1-3B Vision/language No English | |||||
| MIT Not | deployed /leonardo_work/MNRVA_datasets/models/CLIP-ViT-L-14-laion2B-s32B-b82K /gpfs/scratch/shared/ai-hub/models/image-classification/laion/CLIP-ViT-L-14-laion2B-s32B-b82K (restricted) | ||||
| openMaMMUT-ViT-L-14-DataComp-1.4B-s12.8B-b180K 1-3B Vision/language No English | |||||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/openMaMMUT-ViT-L-14-DataComp-1.4B-s12.8B-b180K /gpfs/scratch/shared/ai-hub/models/image-classification/laion/laion/openMaMMUT-ViT-L-14-DataComp-1.4B-s12.8B-b180K (restricted) | |||
| Mistral-7B-v0.3 7-9B Language | (English) No Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Mistral-7B-v0.3 /gpfs/scratch/shared/ai-hub/models/text-models/mistralai/Mistral-7B-Instruct-v0.3 (restricted) | ||
| occiglot-7b-eu5 7-9B Language | (Multilingual) Yes 5 | languages | |||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/occiglot-7b-eu5 /gpfs/ | scratch/shared/ai-hub/models/text-models/occiglot/occiglot-7b-eu5 (restricted) | ||
| Meta-Llama-3.1-8B 7-9B Language | (Multilingual) Yes 8 | languages | |||
| Llama3.1 Not | deployed /leonardo_work/MNRVA_datasets/models/Meta-Llama-3.1-8B /gpfs/ | scratch/shared/ai-hub/models/text-models/meta-llama/Llama-3.1/Llama-3.1-8B (restricted) | |||
| Qwen3-8B-Base 7-9B Language | (Multilingual) No 119 | languages | |||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Qwen3-8B-Base /gpfs/scratch/shared/ai-hub/models/text-models/Qwen/Qwen3/Qwen3-8B-Base (restricted) | |||
| Mistral-Nemo-Base-2407 7-9B Language | (Multilingual) No 12B | ||||
| 9 | languages | ||||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Mistral-Nemo-Base-2407 /gpfs/scratch/shared/ai-hub/models/text-models/mistralai/Mistral-Nemo-Base-2407 (restricted) | |||
| tfree-hat-pretrained-7b-base 7-9B Language | (Multilingual) No License Not | deployed /leonardo_work/MNRVA_datasets/models/tfree-hat-pretrained-7b-base /gpfs/scratch/shared/ai-hub/models/text-models/Aleph-Alpha/Aleph-Alpha/tfree-hat-pretrained-7b-base | |||
| Teuken-7B-base-v0.6 7-9B Language | (Multilingual) No 24 | languages | |||
| cc-by-nc-4.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/openGPT-X/Teuken-7B-base-v0.6 (restricted) | |||
| EuroLLM-9B 7-9B Language | (Multilingual) Yes 35 | languages | |||
| Apache | 2.0 Not | deployed Not | deployed restricted | ||
| salamandra-7b 7-9B Language | (Multilingual) No 35 | languages | |||
| Apache | 2.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/BSC-LT/salamandra-7b (restricted) | ||
| aya-expanse-8b 7-9B Language | (Multilingual) Yes 23 | languages | |||
| cc-by-nc-4.0 Not | deployed Not | deployed restricted | |||
| aya-vision-8b 7-9B Vision/language Yes 23 | languages | ||||
| cc-by-nc-4.0 Not | deployed Not | deployed restricted | |||
| EuroVLM-9B-Preview 7-9B Vision/language No 35 | languages | ||||
| Preview | release | ||||
| Apache | 2.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/utter-project/EuroVLM-9B-Preview (restricted) | ||
| TowerVision-9B 7-9B Vision/language No 18 | languages | ||||
| cc-by-nc-sa-4.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/multimodal-models/utter-project/TowerVision-9B (restricted) | |||
| maya 7-9B Vision/language No 8 | languages | ||||
| 8B | params | ||||
| Instruction-finetuned | |||||
| Apache | 2.0 Not | deployed Not | deployed restricted | ||
| Mistral-Small-3.1-24B-Base-2503 10-30B Language | (Multilingual) No 24 | languages | |||
| Apache | 2.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/mistralai/Mistral-Small-3.1-24B-Base-2503 (restricted) | ||
| Poro-34B 10-30 | B Language | (Multilingual) No English | & | Finnish | |
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Poro-34B /gpfs/scratch/shared/ai-hub/models/text-models/LumiOpen/Poro-34B | |||
| gpt-sw3-20b 10-30 | B Language | (Multilingual) No Nordic | |||
| License Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/AI-Sweden-Model/gpt-sw3-20b (restricted) | |||
| Velvet-14B 10-30 | B Language | (Multilingual) Yes 6 | languages | ||
| Apache | 2.0 Not | deployed Not | deployed restricted | ||
| gemma-3-12b-pt 10-30 | B Language | (Multilingual) Yes 140 | languages | ||
| gemma Not | deployed Not | deployed restricted | |||
| TildeOpen-30b 10-30 | B Language | (Multilingual) No 32 | languages | ||
| Nordic | and | Eastern | EU | ||
| cc-by-4.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/TildeAI/TildeOpen-30b (restricted) | |||
| Viking-13B 10-30 | B Language | (Multilingual) No 5 | languages | ||
| Nordic | |||||
| Apache | 2.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/LumiOpen/Viking-13B (restricted) | ||
| Magistral-Small-2509 10-30 | B reasoning No 24 | languages | |||
| 24B | params | ||||
| Apache | 2.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/mistralai/Magistral-Small-2509 (restricted) | ||
| Pixtral-12B-Base-2409 10-30 | B Vision/language No 9 | languages | |||
| Apache | 2.0 Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/mistralai/Pixtral-12B-Base-2409 (restricted) | |||
| ALIA-40b 40-80 | B Language | (Multilingual) No 35 | languages | ||
| Apache | 2.0 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/BSC-LT/ALIA-40b (restricted) | ||
| Llama-3.1-70B 40-80 | B Language | (Multilingual) Yes 8 | languages | ||
| Llama3 Not | deployed Not | deployed restricted | |||
| Llama-Poro-2-70B-base 40-80 | B Language | (Multilingual) No Finnish/English | |||
| Llama3 Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/text-models/LumiOpen/Llama-Poro-2-70B-base (restricted) | |||
| Mixtral-8x7B-v0.1 40-80 | B Language | (Multilingual) No 5 | languages | ||
| MoE (47B) | |||||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Mixtral-8x7B-v0.1 /gpfs/scratch/shared/ai-hub/models/text-models/mistralai/Mixtral-8x7B-v0.1 (restricted) | |||
| InternVL3-78B-Pretrained 40-80 | B Vision/language No Multilingual | ||||
| License Not | deployed Not | deployed /gpfs/scratch/shared/ai-hub/models/multimodal-models/OpenGVLab/InternVL3-78B-Pretrained (restricted) | |||
| Qwen2-VL-72B 40-80B Vision/language No Multilingual | |||||
| Apache | 2.0 Not | deployed /leonardo_work/MNRVA_datasets/models/Qwen2.5-VL-72B-Instruct /gpfs/scratch/shared/ai-hub/models/multimodal-models/Qwen/Qwen2/Qwen2-VL-72B | (restricted) |
Notes on HPC Centres and Access Conditions
Each participating HPC centre applies its own policies regarding data and model access, storage, and redistribution. As a result, availability and accessibility of datasets and models may vary across systems.
MareNostrum 5 (BSC, Spain): Resources are provided for project use within the centre’s HPC environment. Access is subject to BSC user and data governance policies.
Leonardo (CINECA, Italy): Datasets and models are deployed under EuroHPC and CINECA access rules.
Users are advised to consult the respective centre documentation for detailed access conditions and usage guidelines.