{"id":3505,"date":"2024-05-10T14:14:40","date_gmt":"2024-05-10T21:14:40","guid":{"rendered":"https:\/\/in.nau.edu\/arc\/?page_id=3505"},"modified":"2024-08-14T08:07:42","modified_gmt":"2024-08-14T15:07:42","slug":"apptainer-quickstart","status":"publish","type":"page","link":"https:\/\/in.nau.edu\/arc\/apptainer-quickstart\/","title":{"rendered":"Apptainer &#8211; Quickstart Guide"},"content":{"rendered":"<!-- shortcode-right-column -->\n<div class=\"shortcode-right-column\" >\n    <div class=\"shortcode-right-column__container\"><\/p>\n<p><!-- shortcode-contact -->\n<div class=\"shortcode-contact\">\n    <div class=\"contact-header\">\n        <h3>Advanced Research Computing<\/h3>\n    <\/div>\n    <div class=\"contact-body\">\n                <a href=\"mailto:ask-arc@nau.edu\" aria-label=\"Advanced Research Computing: Email Address\" title=\"Email Address\">\n            <div class=\"contact-icon-container\">\n                <i class=\"fas fa-envelope\" aria-hidden=\"true\"><\/i>\n                <span class=\"sr-only\">Email:<\/span>\n            <\/div>\n            <div class=\"contact-email\">ask-arc&#8203;@nau.edu<\/div>\n        <\/a>\n                    <\/div>\n<\/div>\n\n<\/p>\n<p><\/div>\n<\/div>\n\n<h1 id=\"overview\">Apptainer &#8211; Quickstart Guide<\/h1>\n<p>Apptainer, formerly known as Singularity, is an open-source container platform designed to facilitate the deployment of complex applications on high-performance computing (HPC) clusters. It provides a simple, portable, and reproducible method for running applications, allowing researchers to execute scientific programs without the need for native installation by cluster administrators.<\/p>\n<h2 id=\"understanding-containers\">Understanding Containers<\/h2>\n<h3 id=\"basic-concepts-of-containerization\">Basic Concepts of Containerization<\/h3>\n<p>Containerization is a method of packaging software so that it can run with its dependencies in isolated environments called containers. These containers are lightweight and include everything needed to run the application: code, runtime, system tools, libraries, and settings.<br \/>\nThis technology allows developers to easily deploy applications across different computing environments, from a developer\u2019s laptop to a test environment, from a staging environment into production, and from a physical machine in a data center to a virtual machine in a public or private cloud.<\/p>\n<h3 id=\"difference-between-containers-and-vms\">Difference Between Containers and VM&#8217;s<\/h3>\n<p>Containers and Virtual Machines (VMs) are both technologies used to create efficient, isolated environments for running applications, but they operate differently.<br \/>\nA\u00a0<strong>container<\/strong>\u00a0encapsulates an application\u2019s code along with its libraries, dependencies, and some lightweight operating system APIs and services, all running on the host\u2019s operating system kernel. This makes containers highly portable and efficient, as they are smaller in size and require less overhead than VMs.<br \/>\nOn the other hand, a\u00a0<strong>VM<\/strong>\u00a0includes a full copy of an operating system, including its own kernel, on top of a hypervisor which runs on the physical hardware. This means each VM is entirely isolated with its own resources, but this also results in greater resource consumption compared to containers.<\/p>\n<h3 id=\"use-cases-for-containers-in-scientific-computing\">Use Cases for Containers in Scientific Computing<\/h3>\n<p>Containers have several use cases for scientific computing:<\/p>\n<ul>\n<li>Reproducibility: Containers help ensure that scientific computations are reproducible across different computing environments. By packaging software with all its dependencies, researchers can share their containers with others, knowing that their software will run the same way everywhere.<\/li>\n<li>Collaboration: Containers facilitate collaboration among communities by providing a consistent environment that can be used across various systems. This is particularly useful in large, distributed teams working on complex projects.<\/li>\n<li>Portability: Containers allow applications to be easily moved from one computing environment to another, such as from a local machine to an HPC cluster, without the need for reconfiguration.<\/li>\n<\/ul>\n<h2 id=\"setup\">Setup<\/h2>\n<h3 id=\"loading-apptainer\">Loading Apptainer<\/h3>\n<p>To be able to use the <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">apptainer<\/span> command, you must first load the <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">apptainer<\/span> module.<\/p>\n<pre><code class=\"bash language-bash\">$ module load apptainer<\/code><\/pre>\n<p>To verify that the command was loaded successfully, you can use the following command to see what version of Apptainer is being used.<\/p>\n<pre><code class=\"bash language-bash\">$ apptainer \u2010\u2010version\r\napptainer version 1.2.4+318-g636ab8526<\/code><\/pre>\n<h3 id=\"changing-the-apptainer-cache-directory\">Changing the Apptainer Cache Directory<\/h3>\n<p>By default, Apptainer creates a download cache in the directory <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">$HOME\/.apptainer\/cache<\/span>. For people running Apptainer on a personal PC, that typically isn&#8217;t an issue. However, since the cache directory can grow <em>very<\/em> rapidly, it&#8217;s recommended to move the cache directory to a different directory such as your <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\/scratch<\/span> directory to prevent going over quota.<\/p>\n<p>The <a href=\"https:\/\/apptainer.org\/docs\/user\/1.0\/build_env.html\">Apptainer documentation<\/a> tells us that the bash environment variable <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">APPTAINER_CACHEDIR<\/span> controls where the cache directory is located. To change the cache directory to somewhere such as <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\/scratch\/abc123\/apptainer_cache<\/span> for your current session, you would use the following command:<\/p>\n<pre><code class=\"bash language-bash\">$ export APPTAINER_CACHEDIR=\/scratch\/abc123\/apptainer_cache<\/code><\/pre>\n<p>To make the changes persistent, add the command above to your <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">~\/.bashrc<\/span> file, then load it into your current shell.<\/p>\n<pre><code class=\"bash language-bash\">$ echo \"export APPTAINER_CACHEDIR=\/scratch\/abc123\/apptainer_cache\" &gt;&gt; ~\/.bashrc\r\n$ source ~\/.bashrc<\/code><\/pre>\n<h2 id=\"running-containers-on-the-hpc-cluster\">Running Containers on the HPC Cluster<\/h2>\n<h3 id=\"downloading-containers\">Downloading Containers<\/h3>\n<p>To download a container, you can use the <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">apptainer pull<\/span> command.<\/p>\n<pre><code class=\"bash language-bash\">$ apptainer pull [options] [url]<\/code><\/pre>\n<p>Some common options include:<\/p>\n<ul>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\u2010\u2010dir [directory]<\/span>: Directory to place the downloaded container file. Defaults to the current directory.<\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\u2010\u2010disable-cache<\/span>: Prevents Apptainer from creating cached files of the downloaded container. The purpose of download cache is to speed up downloads by downloading only what&#8217;s not in the cache, and this flag will mean you have to download the files every time you&#8217;re downloading a container.<\/li>\n<\/ul>\n<p>For the <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">[url]<\/span> part of the command, you will need to prepend the container tag with the source of where you&#8217;re downloading your container from. Valid entries listed on the <a href=\"https:\/\/apptainer.org\/docs\/user\/main\/cli\/apptainer_pull.html#apptainer-pull\">Apptainer documentation<\/a> include:<\/p>\n<ul>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">library:\/\/[container:tag]<\/span><\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">docker:\/\/[image:tag]<\/span> (Docker Hub)<\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">shub:\/\/[image:tag]<\/span> (Singularity Hub)<\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">oras:\/\/[image:tag]<\/span> (OCI registry that supports ORAS)<\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">https:\/\/[url]\/[image].sif<\/span> (SIF file from a custom URL)<\/li>\n<\/ul>\n<p>Example:<\/p>\n<pre><code class=\"bash language-bash\">$ apptainer pull --dir \/scratch\/abc123\/ docker:\/\/deeplabcut\/deeplabcut:latest-gui<\/code><\/pre>\n<p><em>Note: If you are downloading a container image that isn&#8217;t in a Singluarity container format (SIF), then Apptainer will automatically convert the image to a SIF image, which may take a bit of time depending on the image.<\/em><\/p>\n<p><em>Note: If you haven&#8217;t configured your cache directory or did not include the <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\u2010\u2010disable-cache<\/span> flag, then your command may be cancelled due to going over your disk quota. See <strong>Changing the Apptainer Cache Directory<\/strong> for instructions on how to change this directory.<\/em><\/p>\n<p>Once you have downloaded the container, you can see that the image has been placed into a file.<\/p>\n<pre><code class=\"bash language-bash\">$ ls \/scratch\/abc123\/\r\ndeeplabcut_latest-gui.sif<\/code><\/pre>\n<h3 id=\"running-a-container\">Running a Container<\/h3>\n<p>Each container may have a specific method to run a given container, so it&#8217;s best that you primarily refer to the documentation for that given image. However, it&#8217;s important to keep in mind that commands for other container solutions such as Docker will not work with Singularity, and will require you to use some intuition and trial-and-error in order to convert the commands and arguments over to Apptainer.<\/p>\n<p>Generally, to run a container, you would use the <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">apptainer run<\/span> command.<\/p>\n<pre><code class=\"bash language-bash\">$ apptainer run [options] [image]<\/code><\/pre>\n<p>There are many possible configuration options available for this command, so you may want to run the command <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">apptainer run \u2010\u2010help<\/span> to view all the available options. However, some general important ones include:<\/p>\n<ul>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">-c<\/span>: Contains the filesystem in the container to a bare minimum. Without the flag, the host OS filesystem is attached to the container.<\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\u2010\u2010fakeroot<\/span>: Runs the container as root. This root user is isolated within the container and contains root privileges <em>only<\/em> for user&#8217;s\/filesystems\/programs inside the container. This is <strong>not<\/strong> a workaround to be able to use <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">sudo<\/span> on Monsoon for anything, but it can be useful for operations such as package management inside a container of an operating system such as Ubuntu.<\/li>\n<li><span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">\u2010\u2010nv<\/span>: Attaches an NVIDIA GPU to the container. This is required for GPU workloads.<\/li>\n<\/ul>\n<p>The <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">[image]<\/span> paramater is interesting in that it can take either a <span style=\"font-size: 16px; font-family: monospace; border: 1px solid; border-radius: 4px; padding: 0px 4px 0px; border-color: #BBBBBB;\">.sif<\/span> image file <em>OR<\/em> a web image link, such as shown in the <strong>Downloading Containers<\/strong> section. If you give it a web image link, then Apptainer will download the image and run it without saving it to the filesystem.<\/p>\n<p>Examples:<\/p>\n<pre><code class=\"bash language-bash\">$ apptainer run --nv \/scratch\/abc123\/deeplabcut_latest-gui.sif\r\n$ apptainer run --nv docker:\/\/deeplabcut\/deeplabcut:latest-gui<\/code><\/pre>\n<h2 id=\"additional-resources\">Additional Resources<\/h2>\n<p>This page only covers the basics on how to get a simple Apptainer container up and running, but there is still much more you can do with it! If you wish to learn more about how to use Apptainer, check out the <a href=\"https:\/\/apptainer.org\/docs\/user\/main\/introduction.html\">Apptainer user documentation<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Apptainer &#8211; Quickstart Guide Apptainer, formerly known as Singularity, is an open-source container platform designed to facilitate the deployment of complex applications on high-performance computing (HPC) clusters. It provides a simple, portable, and reproducible method for running applications, allowing researchers to execute scientific programs without the need for native installation by cluster administrators. Understanding Containers [&hellip;]<\/p>\n","protected":false},"author":2758,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","ring_central_script_selection":"","footnotes":""},"class_list":["post-3505","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/3505","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/users\/2758"}],"replies":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/comments?post=3505"}],"version-history":[{"count":9,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/3505\/revisions"}],"predecessor-version":[{"id":3591,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/3505\/revisions\/3591"}],"wp:attachment":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/media?parent=3505"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}