Need Help? Contact us. help@olcf.ornl.gov
Contact & Support
OLCF User Assistance Center
Authentication Support
OLCF Office Hours
Communication to Users
Accounts and Projects
Request a New Allocation
What are the differences between project types?
What happens after a project request is approved?
Guidance on Frontier Allocation Requests
Applying for a user account
Checking the status of your application
Get access to additional projects
Frequently Asked Questions
How do I apply for an account?
What is the status of my application?
How should I acknowledge the OLCF in my publications and presentations?
What is a subproject?
I no longer need my account. Who should I inform and what should I do with my OLCF issued RSA SecurID token?
My SecurID token is broken/expired. What should I do?
Getting Help
Additional Resources
Documents and Forms
Forms for Requesting a Project Allocation
Forms for Requesting an Account
Forms to Request Changes to Computers, Jobs, or Accounts
Report Templates
Miscellaneous Forms
OLCF Policy Guides
OLCF Acknowledgement
Software Requests
Special Requests and Policy Exemptions
Computing Policy
Computer Use
Data Use
Software Use
User Accountability
Data Management Policy
Introduction
Data Retention, Purge, & Quotas
Data Prohibitions & Safeguards
Software
Security Policy
Scope
Personal Use
Accessing OLCF Computational Resources
Data Management
Sensitive Data
Data Transfer
INCITE Allocation Under-utilization Policy
Project Reporting Policy
Non-proprietary Institutional User Agreement Policy
Access
Rules and Regulations
Safety and Health
Intent to Publish
Export Control
Intellectual Property
HIPAA/ITAR Project Rules of Behavior Policy
User-Managed Software (UMS) Policy
Purpose
Policies
Glossary
Additional Resources
Connecting
Connecting for the first time
Activating a new SecurID fob
PINs, Passcodes, and Tokencodes
X11 Forwarding
Systems Available to All Projects
OLCF System Hostnames
Starting a Tmux Session
Checking System Availability
Systems
2024 Notable System Changes
HPSS Decommission and Kronos Availability
Late July 2024 - Kronos available
August 30, 2024 - HPSS becomes read-only
January 31, 2025 - HPSS decommissioned
Summit and Alpine2 Decommissions
November 15, 2024 - Summit decommissioned
November 19, 2024 - Alpine2 read-only
January 31, 2025 - Alpine2 decommissioned
Frontier User Guide
System Overview
Frontier Compute Nodes
Node Types
System Interconnect
File Systems
Operating System
GPUs
Connecting
Data and Storage
Transition from Alpine to Orion
LFS setstripe wrapper
NFS Filesystem
Lustre Filesystem
Kronos Archival Storage
NVMe
NVMe Usage
Using Globus to Move Data to and from Orion
AMD GPUs
AMD vs NVIDIA Terminology
Blocks (workgroups), Threads (work items), Grids, Wavefronts
The Compute Unit
HIP
Things To Remember When Programming for AMD GPUs
Programming Environment
Environment Modules (Lmod)
Compilers
MPI
Compiling
Compilers
MPI
OpenMP
OpenMP GPU Offload
OpenACC
HIP
HIP + OpenMP CPU Threading
SYCL
Running Jobs
Login vs Compute Nodes
Simplified Node Layout
Slurm
Batch Scripts
Interactive Jobs
Common Slurm Options
Slurm Environment Variables
Job States
Job Reason Codes
Scheduling Policy
Job Dependencies
Monitoring and Modifying Batch Jobs
Srun
Process and Thread Mapping Examples
Ensemble Jobs
Tips for Launching at Scale
Software
Containers
Debugging
Linaro DDT
GDB
Valgrind4hpc
Omnitrace/ROCm Systems Profiler
Profiling Applications
Getting Started with the HPE Performance Analysis Tools (PAT)
Getting Started with HPCToolkit
Getting Started with the ROCm Profiler
Roofline Profiling with the ROCm Profiler
ROCm Compute Profiler (formerly Omniperf)
Tips and Tricks
Using reduced precision (FP16 and BF16 datatypes)
Enabling GPU Page Migration
Floating-Point (FP) Atomic Operations and Coarse/Fine Grained Memory Allocations
Performance considerations for LDS FP atomicAdd()
Library considerations with atomic operations
System Updates
2025-05-13
2025-04-01
2025-02-18
2025-01-14
2024-11-12
2024-09-03
2024-08-20
2024-07-16
2024-04-17
2024-03-19
2024-01-23
2023-12-05
2023-10-03
2023-09-19
2023-07-18
2023-05-09
Known Issues
Open Issues
Open Issues w/Workaround
Resolved Issues
Citadel User Guide
What is Citadel
Citadel (SPI) Documentation
Andes User Guide
System Overview
Compute Nodes
Login Nodes
File Systems
LFS setstripe Wrapper
Shell and Programming Environments
Default shell
Environment Management with lmod
Installed Software
Compiling
Available Compilers
Changing Compilers
Compiler Wrappers
Compiling Threaded Codes
Compiling CUDA Codes
Running Jobs
Login vs Compute Nodes on Commodity Clusters
Slurm
Interactive Batch Jobs on Commodity Clusters
Common Batch Options to Slurm
Batch Environment Variables
Modifying Batch Jobs
Monitoring Batch Jobs
Job Execution
Batch Queues on Andes
Job Accounting on Andes
Debugging
Linaro DDT
GDB
Valgrind
Visualization tools
ParaView
VisIt
Remote Visualization using VNC (non-GPU)
Remote Visualization using VNC (GPU nodes)
Remote Visualization using Nice DCV (GPU nodes only)
Container Usage
Examples for Building and Running Containers
Building a Simple Container Image
Running a Simple Container Through a Batch Job
Running an MPI Application within a Container
Home
System Overview
Access & Connecting
Usage
Acceptable Tasks
Unacceptable Tasks
Data Transfer Nodes (DTNs)
System Overview
Interactive Access
Access From Globus Online
Batch Queue (Slurm)
Queue Policy
Submitting jobs to Frontier
High Performance Storage System
System Overview
Odo
System Overview
File Systems
Obtaining Access to Odo
Logging In to Odo
Spock Quick-Start Guide
System Overview
Spock Compute Nodes
System Interconnect
File Systems
GPUs
Connecting
Data and Storage
NFS
GPFS
Programming Environment
Environment Modules (Lmod)
Compilers
MPI
Compiling
MPI
OpenMP
OpenMP GPU Offload
HIP
Running Jobs
Slurm Workload Manager
Slurm Compute Node Partitions
Process and Thread Mapping
NVMe Usage
Getting Help
Crusher Quick-Start Guide
System Overview
Crusher Compute Nodes
System Interconnect
File Systems
GPUs
Connecting
Data and Storage
NFS Filesystem
Lustre Filesystem
Programming Environment
Environment Modules (Lmod)
Compilers
MPI
Compiling
MPI
OpenMP
OpenMP GPU Offload
HIP
HIP + OpenMP CPU Threading
SYCL
Running Jobs
Slurm Workload Manager
Slurm Compute Node Partitions
Process and Thread Mapping
NVMe Usage
Tips for Launching at Scale
Profiling Applications
Getting Started with the HPE Performance Analysis Tools (PAT)
Getting Started with HPCToolkit
Getting Started with the ROCm Profiler
Roofline Profiling with the ROCm Profiler
Omnitrace
Omniperf
Notable Differences between Summit and Crusher
Using reduced precision (FP16 and BF16 datatypes)
Enabling GPU Page Migration
Floating-Point (FP) Atomic Operations and Coarse/Fine Grained Memory Allocations
Performance considerations for LDS FP atomicAdd()
System Updates
2024-03-19
2024-01-23
2023-12-05
2023-10-03
2023-09-19
2023-07-18
2023-04-05
2022-12-29
Getting Help
Services and Applications
Slate
Overview
What is Slate?
What is Kubernetes?
What is OpenShift?
Getting Started
Requesting A Slate Project Allocation
Logging in
Slate Namespaces
Install the OC tool
Test login with OC Tool
Guided Tutorial
Creating your project
Guided Web GUI Tutorial
Guided Tutorial: CLI
Adding a Pod to your Project
Image Building
Build Types
Examples
Logging into the Registry Externally
Workloads
Pods
Deployments
Networking
Services
NodePorts
Routes
Network Policies
Quick Access from Outside Slate
Persistent Storage
Creating A Persistent Volume Claim
Adding PVC To Pod
Backups
Workflows
Workflows Overview
OpenShift GitOps
OpenShift Pipelines
Application Deployment Examples
Build and Deploy Simple Website
Deploy MongoDB
Deploy NGINX with Hello World
GitLab Runners
Deploy Packages with Helm
Helm Prerequisites
MinIO Object Store (On an NCCS Filesystem)
Access OLCF Resources From Containers
Batch Job Submission
Mount OLCF Filesystems
Schedule Other Slate Resources
GPUs
OLCF-Provided Applications on Slate
Troubleshooting
Fix Container Image Permissions
Debugging
YAML Object Quick Reference
CronJobs
Deployments and Stateful Sets
Pods
Roles and Rolebindings
Routes, Services and Nodeports
Persistent Volume Claims
Glossary
myOLCF
Overview
What is myOLCF?
What can it do?
Can I suggest a feature?
Authenticating
OLCF Moderate Accounts
OLCF Open Accounts
Project Pages
Project Context
Switching Project Contexts
Available Pages
Account Pages
Account Context
Available Pages
Processing Project Membership Requests
Jupyter
Overview
Jupyter at OLCF
Access
CPU vs. GPU JupyterLab (Available Resources)
Working within Lustre and NFS (Launching a Notebook)
Conda Environments and Custom Notebooks
Manually Stopping Your JupyterLab Session
Things to Be Aware Of
Example Jupyter Notebooks
Data Storage and Transfers
Summary of Storage Areas
Notes on User-Centric Data Storage
User Home Directories (NFS)
Notes on Project-Centric Data Storage
Project Home Directories (NFS)
Project Work Areas
Project Archive Directories
Data Policies
Information
Special Requests
Data Retention
Orion Lustre HPE ClusterStor Filesystem
Orion Performance Tiers and File Striping Policy
I/O Patterns that Benefit from File Striping
LFS setstripe wrapper
Lustre File Locking Tips
Darshan-runtime and I/O Profiling
Purge
HPSS Data Archival System
Kronos Nearline Archival Storage System
Access / Data Transfer
Directory Structure
Project Quotas
Kronos and HPSS Comparison
Transferring Data
Globus
Using Globus to Move Data Between Collections
Using Globus From Your Local Workstation
HSI
Additional HSI Documentation
HTAR
HTAR Limitations
Additional HTAR Documentation
Command-Line/Terminal Tools
Software
Software News
Frontier: Core Module (March 18, 2025)
Frontier: System Software Update (February 18, 2025)
Frontier: Updated Modules for cpe/23.12 (October 16 2024)
Frontier: Core Module (October 15, 2024)
Frontier: System Software Update (July 16, 2024)
Frontier: User Environment Changes (July 9, 2024)
ML/DL & Data Analytics
Apache Spark
Overview
Getting Started
PyTorch on Frontier
Table of Contents:
Installing PyTorch
Best Practices
PyTorch Geometric
Troubleshooting
Additional Resources
JAX
Setting up the environment
Installing JAX
Testing JAX
Additional Resources
Python on OLCF Systems
Overview
OLCF Python Guides
Conda Basics
Installing mpi4py and h5py
Installing CuPy
Sbcast Conda Environments
Jupyter Visibility
PyTorch on Frontier
JAX
Module Usage
Base Environment
Custom Environments
How to Run
Frontier / Andes
Best Practices
Additional Resources
Profiling Tools
Score-P
Overview
Usage
Instrumentation
Measurement
Profiling
Tracing
Manual Instrumentation
Score-P Demo Video
Tuning and Analysis Utilities (TAU)
Enabling TAU at OLCF
Profile and trace using “tau_exec exe”
Automatic source instrumentation using compiler wrappers
Selective Instrumentation
Manual source instrumentation
Run-Time Environment Variables
Compile-Time Environment Variables
References
Vampir
Overview
Usage
Vampir on a Login Node
Vampir Using VampirServer
Vampir Tunneling to VampirServer
Vampir GUI Demo
User-Managed Software
Introduction
Currently Available User-Managed Software
Usage
Policies
Writing UMS Modulefiles
Workflows
Running Workflows on OLCF Resources
Workflow Systems
Ensemble Toolkit (EnTK)
Parsl
Swift/T
Visualization Tools
VisIt
Overview
Installing and Setting Up Visit
Remote GUI Usage
Command Line Example
Troubleshooting
Additional Resources
ParaView
Overview
Installing and Setting Up ParaView
Remote GUI Usage
Command Line Example
Troubleshooting
Additional Resources
Containers on Frontier
Examples for Building and Running Containers
Building and running a container image from a base Linux distribution for MPI
Pushing your Apptainer image to an OCI Registry supporting ORAS (e.g. DockerHub)
Building an image on top of an existing image (local, docker image, OCI artifact)
OLCF Base Images & Apptainer Modules
Base Images
Apptainer Modules
Example Workflow
Sample Applications
Some Restrictions and Tips
Debugging
Linaro Forge DDT
Client Setup and Usage
Download
Installation
Configuration
GNU GDB
Valgrind
Training
OLCF Training Calendar
OLCF Tutorials
OLCF Training Archive
OLCF GPU Hackathons
FacultyHack
OLCF Vimeo Channel
New User Quick Start
Quantum
Quantum Computing User Program (QCUP) Access
QCUP Priorities
Project Allocations
What happens after a project request is approved?
Project Renewals
Closeout and Quarterly Reports
User Accounts
Checking the status of your application
Accessing Quantum Resources
IBM Quantum Computing
Quantinuum
IonQ
IQM
Publication Citations
Quantum Systems
IBM Quantum
Overview
Connecting
Running Jobs & Queue Policies
Checking System Availability & Capability
Software
Additional Resources
Quantinuum
Overview
Connecting
Running Jobs & Queue Policies
Default Quotas
Allocation Policies & Credit Usage
Software
Additional Resources
IonQ
Overview
IonQ systems
Connecting
Running Jobs & Queue Policies
Allocations & Credit Usage
Checking System Availability & Capability
Additional Resources
IQM
Overview
Connecting
Running Jobs & Queue Policies
Checking System Availability
Software
Additional Resources
Quantum Software
Quantum Software on HPC Systems
Overview
Qiskit
PyQuil/Forest SDK (Rigetti)
PennyLane
Pytket
CUDA-Q
Batch Jobs
Hello QCUP Scripts
Overview
IBM Quantum
Quantinuum
IonQ
IQM
Frequently Asked Questions
How do quantum computers differ from classical computers?
What is a qubit?
How do I access the OLCF quantum computing resources?
What happens after I apply for access to QCUP?
I formerly had access to quantum resources, but my backends/lattices/etc. have disappeared, what do I do?
I applied to a quantum computing resource via the vendor website, but don’t have access; what do I do?
Scalable Protected Infrastructure (SPI)
What is SPI
What is Citadel
New User QuickStart
Notable Differences
Allocations and User Accounts
Allocations (Projects)
Requesting a New Allocation (Project)
User Accounts
Requesting a New User Account
Available Resources
Compute
File Systems
Data Transfer
IP Whitelisting
Whitelisting an IP or range
Finding your IP
Citadel
Login Nodes
Connecting
Building Software
External Repositories
Running Batch Jobs
File Systems
Data Transfer
Advanced Computing Ecosystem Testbed (ACE)
Defiant Quick-Start Guide
Defiant System Decommission
System Overview
Defiant Compute Nodes
System Interconnect
File Systems
GPUs
Connecting
Data and Storage
NFS Filesystem
Lustre Filesystem (Polis)
Programming Environment
Environment Modules (Lmod)
Compilers
MPI
Compiling
MPI
OpenMP
OpenMP GPU Offload
HIP
Running Jobs
Slurm Workload Manager
Slurm Compute Node Partitions
Process and Thread Mapping
NVMe Usage
Container Usage
Setup before Building
Build and Run Workflow
Getting Help
Known Issues
Contributing to these docs
Submitting suggestions
Authoring content
Setup authoring environment
Edit the docs
Resources
GitHub Guidelines
OLCF User Documentation
Systems
Edit on GitHub
|
OLCF Home Page
Systems
2024 Notable System Changes
HPSS Decommission and Kronos Availability
Summit and Alpine2 Decommissions
Frontier User Guide
System Overview
Connecting
Data and Storage
Using Globus to Move Data to and from Orion
AMD GPUs
Programming Environment
Compiling
Running Jobs
Software
Containers
Debugging
Profiling Applications
Tips and Tricks
System Updates
Known Issues
Citadel User Guide
What is Citadel
Andes User Guide
System Overview
Shell and Programming Environments
Compiling
Running Jobs
Debugging
Visualization tools
Container Usage
Home
System Overview
Access & Connecting
Usage
Data Transfer Nodes (DTNs)
System Overview
Interactive Access
Access From Globus Online
Batch Queue (Slurm)
High Performance Storage System
System Overview
Odo
System Overview
Spock Quick-Start Guide
System Overview
Connecting
Data and Storage
Programming Environment
Compiling
Running Jobs
Getting Help
Crusher Quick-Start Guide
System Overview
Connecting
Data and Storage
Programming Environment
Compiling
Running Jobs
Profiling Applications
Notable Differences between Summit and Crusher
System Updates
Getting Help