Background

Recent performance breakthroughs in artificial intelligence (AI) and machine learning (ML), including advances in deep learning (DL) and the availability of powerful, easy-to-use ML toolboxes, have led to unprecedented interest in AI/ML among nuclear engineers. However, the full potential of AI/ML techniques has yet to be fully realised. One barrier is that existing ML methods often do not meet the needs of nuclear engineering (NE) applications. Application-agnostic algorithms, or those designed for more traditional ML applications such as computer vision and natural language processing, cannot typically be directly applied to scientific data in NE and require non-trivial, task-specific modifications. Furthermore, there is no common AI/ML benchmark or activity to validate and compare different developed ML methods and algorithms. Finally, there are significant gaps in the predictive capability assessment and improvement of ML models. Specifically, to enable more trustworthy applications in high-consequence systems like nuclear reactors that are subject to nuclear safety regulations, the ML practitioners have to consider a few critical unresolved issues, including the verification, validation and uncertainty quantification (VVUQ) of AI/ML, data scarcity, scaling-induced uncertainty, and lack of physics in black-box ML models. All this requires the development of an AI/ML benchmark that will provide guidelines and exercises in various computational domains to help participants to develop and evaluate the performance of their ML methods.

Objectives

The Task Force (TF) will design benchmark specifications to evaluate the performance of AI/ML in multi-physics modeling and simulation (M&S) of reactor systems, under the guidance of the Expert Group on Reactor Systems Multi-Physics (EGMUP), will provide guidelines to the Working Party on Scientific issues and Uncertainty Analysis of Reactor Systems (WPRS), and the nuclear community on the scientific development needs (data, methods, and benchmark exercises) for trustworthy AI/ML applications to nuclear scientific computing problems. This benchmark will focus on establishing and improving ML predictive capability/credibility, through consistent, comprehensive, and rigorous VVUQ of AI/ML which matches the quality standards for VVUQ of traditional nuclear M&A. Innovative methodologies for unresolved issues such as data scarcity, scaling-induced uncertainty, and lack of physics in black-box ML models will be developed through the benchmark exercises. Finally, the participants will also contribute to enhancing the trustworthiness of AI/ML for nuclear applications in this benchmark, in order to enable wide acceptance of AI/ML by the nuclear regulators, stakeholders, and policy and decision makers. Among the broad concepts incorporated in AI/ML trustworthiness, this benchmark will focus on accuracy, robustness (reproducibility, applicability) and transparency (explainability, interpretability).

This TF will design and supervise the execution of benchmark exercises on both single physics (reactor physics, thermal-hydraulics, fuel performance and structural mechanics) and multi-physics coupled simulation problems. The exercises will target specific challenges of each computational domain. Certified experimental data, as well as verified and validated high-fidelity computational data from other WPRS expert groups will be leveraged as training dataset for benchmark exercises. Both steady-state and time-dependent problems will be included. A broad spectrum of AI/ML sub-domains will be explored, including supervised ML, unsupervised ML, semi-supervised ML, deep generative learning, and probabilistic ML. In each exercise, participants are asked to select AI/ML algorithms to perform tasks specified in the exercises (prediction, generalisation, uncertainty quantification, etc.). To fulfil the above, this TF will provide the following:

• Standardised benchmark exercises with certified experimental data and high-fidelity computational data for the training of AI/ML models;
• Detailed guidelines for applying AI/ML methodologies for supervised, unsupervised, and semi-supervised ML, as well as advanced topics such as deep generative learning and probabilistic ML;
• Proposals towards the development of VVUQ requirements of AI/ML models in nuclear systems based on consensus positions of the TF;
• Guidelines for improving AI/ML trustworthiness through accuracy, robustness (reproducibility, applicability) and transparency (explainability, interpretability);
• Training opportunities to demonstrate AI/ML principles and practices; and
• Demonstrations of the AI/ML guidelines for specific applications.

Scope

Under the guidance of the WPRS, and with contributions of the Expert Group on Physics of Reactor Systems (EGPRS), the Expert Group on Reactor Core Thermal-Hydraulics and Mechanics (EGTHM), and the Expert Group on Reactor Fuel Performance (EGRFP), this TF will promote the use of AI/ML to make impact on solutions of critical issues in nuclear engineering and accelerate research on AI/ML for advanced reactor design and safety/licensing analysis. The TF will develop and supervise benchmark exercises and will establish comprehensive, systematic, and quantifiable VVUQ guidelines and metrics to establish AI/ML predictive capability by building trustworthy ML models with considerations of AI/ML accuracy, robustness (reproducibility, applicability) and transparency (explainability, interpretability). The guidelines will be developed and demonstrated on several benchmark exercises that consider the following:

• Steady-state problems for single physics problems that make use of supervised ML (e.g., regression and classification);
• Transient, time-dependent problems for single physics problems that also make use of unsupervised ML (e.g., dimensionality reduction, clustering);
• Multi-physics problems with steady-state (and potentially transient) scenarios;
• Problems that involve uncertainty quantification of AI/ML models, such as neural networks with various approaches (sampling-based, ensele-based, probabilistic ML-based, etc.);
• Problems with limited data for investigation of deep generative learning;
• Optimisation problems for developing reinforcement learning methods or other types of ML techniques.

The AI/ML benchmarks will be organised as exercises based on the specific AI/ML task and each exercise will contain test cases for various computational domains of interest that will span neutronics, thermal-hydraulics, fuel performance and multi-physics:

• Exercise I: Regression, classification and VVUQ;
• Exercise II: Dimensionality reduction and anomaly detection;
• Exercise III: Generative deep learning and data augmentation;
• Exercise IV: Design optimisation.

Challenging problems using real-world measurement data from nuclear industry will also be considered. Data for these problems will be made available at OECD/NEA GitLab repositories and made accessible to the benchmark participants. Participants are encouraged to propose state-of-the-art AI/ML-based solutions to these problems, thus significantly promoting the use of innovative AI/ML to make impact on solutions of critical issues in nuclear industry and accelerate the AI/ML research and development for advanced reactor design and safety/licensing analysis.

The development of the benchmark specifications will be conducted in two phases. The first phase of the benchmark will focus on Exercise I and II while the second phase on Exercise III and IV.

Active Benchmark Exercises

• October 2023 - December 2024: Phase 1 of the Critical Heat Flux Exercises  

Working Methods

Participants will meet once per month during the first year, and bimonthly in the later phase.

The TF is supervised by the EGMUP, and the co-ordinators report on the TF progress during the EGMUP annual meetings. Short courses on AI/ML training, as well as workshops on the latest advancement of AI/ML for nuclear scientific computing will be organised as part of the WPRS workshops and education activities.

Organisation

Co-ordinators: Xu WU, Gregory DELIPEI (North Carolina State University, USA)

NEA Secretariat: Oliver BUSS

MyNEA Working Area

Participation

Participation is open to all NEA member states. Please contact the NEA Secretariat to join the Task Force.

Schedule