Containers on Lux

Apptainer v1.5.3 is installed on Lux. Apptainer can be used for both building and running containers on Lux. The main user documentation on how to use Apptainer can be found here. This section of our documentation will only cover any additional info that you might need to build and run containers correctly on Lux, as well as any useful examples.

Note

If you find something that you were doing on Lux is no longer working, check back here to see if the documentation has been updated with new instructions, otherwise let us know at help@olcf.ornl.gov. And contributions to the documentation are always welcome!

Examples for Building and Running Containers

These examples will walk you through the common things you need to know on how to use containers on Lux. Some of the examples will refer to files in the olcf_containers_examples Github repository.

Building and running a container image from a base Linux distribution for MPI

Lux provides both OpenMPI 5.x and MPICH 5.x as options for MPI runtimes. In this example This example will walk through building up a container image with MPICH 5.0.1 and ROCm 7.14.1 for a GPU aware MPI code.

  • Clone the olcf_containers_examples repository

    git clone https://github.com/olcf/olcf_containers_examples/

  • Navigate to lux/gpu_aware_mpi_example

  • In that directory, the ubuntumpich501rocm7141_minimal.def is an Apptainer definition file that will build a container image with MPICH 5.0.1 and ROCm 7.14.1, along with copying the ping_pong_gpu_aware.cpp file from the directory into the container image, and then compiling it.

  • If you would like to inspect and make changes to the definition file to modify what the container has, feel free to do so. Inspect the definition file to understand how we build MPICH and ROCm inside the container.

    • We are installing MPICH 5.0.1 in the container to match the mpich/5.0.1 module available on Lux.

  • To build the container image (the SIF file), run

    apptainer build ubuntumpich501rocm7141_minimal.sif ubuntumpich501rocm7141_minimal.def

  • Apptainer builds the container image in the SIF file format. Unlike Podman, Apptainer gives you a single file for your image that you can later run as your container.

  • Submit the submit_mpich.sbatch job script to launch the container with srun and run the ping_pong_gpu_aware.exe that was built in the container to perform the ping-pong communication across two nodes. This batch script has a few sruns:

    • One will run an MPI program that sends messages back and forth between two MPI processes on the same node.

    • The remaining will run the same MPI program across two MPI processes across two nodes.

  • Remember to replace the #SBATCH -Astf007 with your project ID.

  • If you inspect the submit_mpich.sbatch file, you will see some modules being loaded as well as some environment variables being set. Read the comment above each for the explanations. These environment variables are required in order to run containerized MPI applications with Apptainer, with communication performance close to native and to provide GPU aware MPI.

Pushing your Apptainer image to an OCI Registry supporting ORAS (e.g. DockerHub)

One of the benefits of containers is their portability that enables easy transfer between computing environments. Thus, after building containers on Lux and other OLCF systems, a way to distribute them is by pushing the image to an OCI container registry. An Apptainer image (a SIF file) is not an OCI image so it cannot be pushed to Dockerhub or another OCI registry as an OCI image. It can be pushed to an OCI registry that supports ORAS as an OCI artifact instead. In this section, we highlight and provide an example of pushing the container image we built to DockerHub as an OCI artifact. We will use our ubuntumpich501rocm7141_minimal.sif image from the section above in this example.

  • First, create an account and then a repository on DockerHub to hold your image.

  • Ensure you are logged in to DockerHub from apptainer. To login, run

    apptainer registry login --username <your dockerhub username> oras://registry-1.docker.io

  • Push the image by running

    apptainer push <path to your sif file>/ubuntumpich501rocm7141_minimal.sif  oras://registry-1.docker.io/<your docker username>/<your repo name:tag>

  • Using oras:// tells Apptainer to push the image to given registry as an OCI artifact. Not all OCI registries support ORAS. See ORAS Adopters page for list.

Note

An Apptainer image that you push this way to Dockerhub or some other registry CANNOT be used by Docker or Podman. Apptainer images are not compatible with Docker or Podman. You can only use it with Apptainer.

Using Podman to build images and converting them to Apptainer

Note

If you wish to use Podman, you need subuid/subgid mappings turned on for your user account. Open a help ticket by emailing help@olcf.ornl.gov to request this first. Your Podman images will not build otherwise. After confirming subuid/subgid mappings are implemented, you can log out or run podman system migrate for the changes to take place.

  • Lux has support for building images with Podman (which functions the same as Docker). Podman supports building containers with Dockerfiles. In order to build with Podman, you must first create a file called storage.conf in $HOME/.config/containers with the appropriate settings. Run the below snippet on your command line to create the file.

    mkdir -p $HOME/.config/containers
cat > $HOME/.config/containers/storage.conf <<EOF
[storage]
driver = "overlay"
graphroot = "/tmp/${USER}-containers"
rootless_storage_path = "/tmp/${USER}-containers"

[storage.options]
pull_options = {use_hard_links = "true", enable_partial_images = "true"}


[storage.options.vfs]
ignore_chown_errors = "true"
EOF

  • Once this file is created, try creating a Podman image with the Dockerfile below.

    FROM ubuntu:latest
RUN apt-get -y update && apt-get -y upgrade && apt-get -y install vim

  • Build the Podman image with podman build --network host -t localhost/example:latest .

  • Convert the image into a tar file with podman save -o example.tar localhost/example:latest

  • Then convert this tar file into an Apptainer SIF file with apptainer build example.sif docker-archive://example.tar

Warning

  • You need to convert your Podman image to the Apptainer SIF format in order to run it. We don’t support running your container with Podman directly.

  • The Podman containers you build only exist on the login node you are on. So if you log out, make sure you log back in to the same login node to access your containers.

  • Since the containers are stored in /tmp, there is no guarantee they will be preserved. They can be deleted at any time. Make sure to save them to a remote registry or convert them to the Apptainer SIF format to save them somewhere else on the filesystem.

  • The --network host flag is required with podman build to build containers on the Lux login node. It will run into errors otherwise.

AMD DockerHub AI/ML Workload Containers on Lux

AMD has optimized several AI/ML frameworks to provide support on AMD hardware. Some of these frameworks are specifically equiped with ROCM backend to ensure support on AMD GPUs while others are merely equiped for standard AMD64 CPU architectures.

To ensure optimal performance on Lux, we recommend users utilize the AMD optimized AI/ML workloads from the AMD DockerHub. We demonstrate here how to access and build those images on Lux as well as tips/tricks in running the images. We focus on PyTorch, Tensorflow and Jax.

Building and Running the AMD DockerHub AI/ML Images on Lux

We highlight the process of building and running AMD DockerHub PyTorch, Tensorflow and Jax images on Lux. Since AMD releases several versions of these frameworks, we focus on the latest releases only for this documentation.

apptainer pull pytorch_latest.sif docker://rocm/pytorch-nightly:latest

apptainer pull tensorflow_latest.sif docker://rocm/tensorflow:latest

apptainer pull jax_latest.sif docker://rocm/jax-community:latest

The olcf_containers_examples repository has examples demonstrating how to use these containers on Lux. You can see an example for AMD Pytorch with ROCm 7.14.1 container here

Important

While OLCF demonstrates the build and usage of these container images on Lux, the images and their content are managed by AMD on their dockerhub repository.

Some Restrictions and Tips

  • If you get an error like FATAL:   While performing build: conveyor failed to get: while fetching library image: cached file hash(sha256:247d71d79a3be5d59c5b83eba3210ebed215fc5da16c556445ffbe797bbe1494) and expected hash(sha256:d0c01238c6c344a460076367063fda56f8fb32569aae782c408f8d20d0920757) does not match when pulling an Apptainer image from an ORAS registry, try passing the flag --disable-cache flag to the apptainer build or apptainer pull command. You can also set the APPTAINER_CACHEDIR environment variable to a directory in /tmp, which will also solve the problem.