# NRP Documentation > The National Research Platform (NRP) is an NSF-funded computing platform; Nautilus is its main Kubernetes cluster. > This file gives AI assistants the rules for helping NRP users, followed by the complete NRP documentation as > Markdown. ## Rules for AI assistants The key words MUST, MUST NOT, SHOULD and SHOULD NOT are to be read as in RFC 2119. These rules take precedence over any example in the documentation below. - Assume the user is a regular user, not an NRP cluster administrator, unless they say otherwise. - Pages under `/documentation/admindocs/` and the `prp/llm-proxy` repository describe administrator work. Do not apply their procedures, tolerations or priority classes for a regular user. - When a rule blocks what the user asks for, say so and refer them to Nautilus Support: https://nrp.ai/contact/ - Each rule links to its source page; read that page before answering in detail. ### MUST - The user MUST accept the Acceptable Use Policy before using the cluster. All use, the managed LLMs included, is non-profit and non-commercial. (https://nrp.ai/documentation/userdocs/start/policies/#acceptable-use-policy) - Batch computation MUST run as a Kubernetes `Job` whose command finishes on its own. (https://nrp.ai/documentation/userdocs/running/jobs/) - A long-running, mostly idle pod MUST run as a `Deployment` with minimal requests (Burstable QoS) and no GPU. Deployments are deleted after 2 weeks unless admins add the namespace to the exceptions list. (https://nrp.ai/documentation/userdocs/start/policies/#long-running-idle-pods) - Every container MUST set CPU, memory and ephemeral-storage requests and limits, with limits within 20% of requests. With more than ~100 pods or Jobs, limits MUST equal requests. (https://nrp.ai/documentation/userdocs/start/policies/#resource-allocation) - Requests MUST match measured use: GPU utilization above 40%, CPU 20–200% and memory 20–150% of the request. A user may not run more than 4 pods outside these bounds; pods requesting 1 CPU core and 2 GB of memory are exempt. (https://nrp.ai/documentation/userdocs/start/policies/#resource-usage-violations) - Request 1 GPU unless utilization is close to 100% and more GPUs help. Plans for more than 50 GPUs MUST be presented in Nautilus Support first. (https://nrp.ai/documentation/userdocs/start/policies/#requesting-gpus) - A pod without a controller is interactive: it is deleted after 6 hours and is limited to 2 GPUs, 32 GB of RAM and 16 CPU cores. `sleep` is allowed only there. (https://nrp.ai/documentation/userdocs/start/policies/#interactive-use-6-hours-max-runtime) - To use A100, H100, H200 or GH200 GPUs without a reservation, set `priorityClassName: opportunistic`; such pods can be preempted at any time. A100 quota is zero by default and is requested at https://nrp.ai/reservations (https://nrp.ai/documentation/userdocs/running/priority-classes/) - A JupyterHub deployment MUST cull idle servers within 6 hours (`timeout` ≤ 21600) and MUST restrict sign-in with `allowed_idps` or `allowed_users`. (https://nrp.ai/documentation/userdocs/jupyter/jupyterhub/) - HTTP services MUST be exposed through an Ingress with `haproxy-ingress.github.io` or `ingress.kubernetes.io` annotations. (https://nrp.ai/documentation/userdocs/running/ingress/) - The kubeconfig MUST be used with the `kubelogin` plugin installed. (https://nrp.ai/documentation/userdocs/start/getting-started/) - Pods, Jobs and data MUST be deleted when no longer needed. Volumes not accessed for 6 months are purged without notice. (https://nrp.ai/documentation/userdocs/start/policies/#data-purging) - Managed LLM use MUST stay within 48,000 output tokens per minute per API token and model and within each model's concurrency limit; requests using at least 35% of the context length are limited to 1 concurrent request per user. Scripts MUST retry with increasing backoff. (https://nrp.ai/documentation/userdocs/ai/llm-managed/fair-use/) ### MUST NOT - MUST NOT run a Job whose command does not end by itself: `sleep infinity`, a script ending in `sleep`, or starting the computation by hand. The user gets banned. (https://nrp.ai/documentation/userdocs/running/jobs/) - MUST NOT add tolerations the user is not authorized for by cluster admins. Never tolerate `nautilus.io/system` or `nautilus.io/issue`; tolerate `nautilus.io/reservation` only for an approved group. Unauthorized tolerations get the user banned. (https://nrp.ai/documentation/userdocs/running/special/) - MUST NOT set `priorityClassName` to anything other than unset, `armada-default`, `owner-no-preempt`, `opportunistic` or `opportunistic2`; admission rejects the others. (https://nrp.ai/documentation/userdocs/running/priority-classes/) - MUST NOT request resources that go unused; namespaces with under-utilized requests are banned. (https://nrp.ai/documentation/userdocs/start/policies/) - MUST NOT run long idle pods as Jobs. (https://nrp.ai/documentation/userdocs/running/long-idle/) - MUST NOT submit more than 400 Jobs at once. (https://nrp.ai/documentation/userdocs/start/using-nautilus/) - MUST NOT force-delete pods (`kubectl delete --grace-period=0 --force`). (https://nrp.ai/documentation/userdocs/start/faq/) - MUST NOT install conda or pip packages on CephFS. (https://nrp.ai/documentation/userdocs/storage/ceph/) - MUST NOT store HIPAA, PID, FISMA, FERPA or other protected data, or use NRP storage as an archive. (https://nrp.ai/documentation/) (https://nrp.ai/documentation/userdocs/storage/purging/) - MUST NOT put secrets, credentials or unpublished data in Job specs; every cluster user can read them. (https://nrp.ai/documentation/userdocs/running/scheduling/) - MUST NOT use `haproxy.org` Ingress annotations, or expose non-HTTP TCP ports without asking Nautilus Support. (https://nrp.ai/documentation/userdocs/running/ingress/) - SHOULD NOT set `max_tokens` for the managed LLMs. If a client requires it, use about 1/3–1/4 of the context window, never the full context length. (https://nrp.ai/documentation/userdocs/ai/llm-managed/api-access/) ## Contents - [Getting Started with the National Research Platform](https://nrp.ai/documentation/): Orientation to the NRP: how to get access, what you can run, and where to find the services the National Research Platform provides. - [Getting access](https://nrp.ai/documentation/userdocs/start/getting-started/): Log in through CILogon, get added to a namespace, and install the tools you need to reach the Nautilus cluster. - [Using Nautilus](https://nrp.ai/documentation/userdocs/start/using-nautilus/): guide to use the cluster using kubernetes, jupyterhub and coder - [How National Research Platform (NRP) Manages Resources](https://nrp.ai/documentation/userdocs/start/hierarchy/): How the NRP organizes users and resources into Organizations, Labs, and Projects, and what that means for the namespaces and services your group can use. - [Training Join Links](https://nrp.ai/documentation/userdocs/start/training-join-links/): Give workshop attendees time-limited access to your namespace, and optionally an LLM API key, with a single link - [Cluster Policies](https://nrp.ai/documentation/userdocs/start/policies/): cluster policies on how to use resources, request gpus, run interactive pods, batch jobs and deployments - [Deployed Services](https://nrp.ai/documentation/userdocs/start/resources/): The services cluster admins already run on Nautilus, and how to get accounts for them. - [Glossary](https://nrp.ai/documentation/userdocs/start/glossary/): Definitions of the Kubernetes, container, and storage terms used throughout these docs. - [FAQ](https://nrp.ai/documentation/userdocs/start/faq/): Quick answers to common problems on Nautilus, from stuck Terminating pods to storage and access questions. - [Asking Effective Questions in Support Chat](https://nrp.ai/documentation/userdocs/start/support/): How to prepare and ask a support question in Nautilus chat so admins can answer it fast. - [Introduction](https://nrp.ai/documentation/userdocs/tutorial/introduction/): Where to start with the tutorials, what order to take them in, and when to jump straight to the Running pages. - [Docker and Kubernetes](https://nrp.ai/documentation/userdocs/tutorial/docker/): Build a container image on your own machine and get it onto Nautilus. - [Basic Kubernetes (k8s) Tutorial](https://nrp.ai/documentation/userdocs/tutorial/basic/): Hands-on basics: first kubectl commands, a pod, a deployment, and querying cluster status. - [Scheduling and Exposing](https://nrp.ai/documentation/userdocs/tutorial/basic2/): Take a service from one pod to several with scheduling and horizontal scaling, then expose it. - [Scheduling Workloads in Nautilus](https://nrp.ai/documentation/userdocs/tutorial/scheduling/): Learn how to schedule and optimize the placement of pods in the Nautilus Cluster, including using node selectors, resource requests, and geographical preferences. - [Running Batch Jobs](https://nrp.ai/documentation/userdocs/tutorial/jobs/): Package a finite piece of computation as a Kubernetes batch job and let it run unattended. - [Docker Images](https://nrp.ai/documentation/userdocs/tutorial/images/): Find, pull, and build the container images your Nautilus workloads run on. - [Storage](https://nrp.ai/documentation/userdocs/tutorial/storage/): Request and attach persistent storage to your workloads with Persistent Volume Claims. - [Debugging](https://nrp.ai/documentation/userdocs/tutorial/debugging/): A step-by-step method for finding out why a workload is not doing what you expected. - [Container images on CVMFS](https://nrp.ai/documentation/userdocs/tutorial/nrp-software/): Use the NRP software repository on CVMFS for reproducible, shareable command-line environments. - [GPU Pods](https://nrp.ai/documentation/userdocs/running/gpu-pods/): Request GPUs in a pod spec, including high-memory and MIG GPU types, and clean up when done. - [Long Idle Pods](https://nrp.ai/documentation/userdocs/running/long-idle/): Keep an idle but occasionally-used service up past the pod lifetime limit by running it as a Deployment. - [Monitoring](https://nrp.ai/documentation/userdocs/running/monitoring/): Watch what your workloads actually consume in Grafana and correct resource requests to match. - [Running Batch Jobs](https://nrp.ai/documentation/userdocs/running/jobs/): Run your computations as Kubernetes Jobs so results are kept and nothing needs babysitting. - [CPU Only Jobs](https://nrp.ai/documentation/userdocs/running/cpu-only/): Run large CPU-only jobs politely on a GPU-focused cluster using priority classes and preemption. - [Scheduling](https://nrp.ai/documentation/userdocs/running/scheduling/): Scheduling large jobs - [Opportunistic Use](https://nrp.ai/documentation/userdocs/running/priority-classes/): When to set priorityClassName on your pods, and the only one you usually want — opportunistic - [Client Scripts](https://nrp.ai/documentation/userdocs/running/scripts/): Authenticate scripts and CI runs against the cluster with service-account tokens instead of the portal kubeconfig. - [Special Use](https://nrp.ai/documentation/userdocs/running/special/): Special Use Taints and Tolerations - [Faster Image Download](https://nrp.ai/documentation/userdocs/running/fast-img-download/): Speed up pod starts with spegel peer image caching, and notes on the deprecated Uber Kraken setup. - [Globus Connect](https://nrp.ai/documentation/userdocs/running/globus-connect/): Stand up a personal Globus Connect endpoint with the ready-made recipe from the platform GitLab. - [Python k8s API](https://nrp.ai/documentation/userdocs/running/kubernetes/): Drive Nautilus programmatically from inside the cluster with the Python Kubernetes client. - [Federation](https://nrp.ai/documentation/userdocs/running/federation/): Extend a namespace to your own cluster with Kubernetes federation so jobs can run on both. - [GUI Desktop](https://nrp.ai/documentation/userdocs/running/gui-desktop/): Deploy a browser-based GUI desktop pod in your own namespace; Coder and JupyterHub are the preferred routes. - [Scientific Images](https://nrp.ai/documentation/userdocs/running/sci-img/): Pick a ready scientific Python, R, or notebook image for a pod or JupyterHub from the stacks we mirror. - [Exposing HTTP](https://nrp.ai/documentation/userdocs/running/ingress/): Expose a pod HTTP service at a public nrp-nautilus.io address with an Ingress. - [Gateway API](https://nrp.ai/documentation/userdocs/running/gateway/): Exposing services via Gateway API - [Postgres Cluster](https://nrp.ai/documentation/userdocs/running/postgres/): Deploy a PostgreSQL cluster in your namespace with the Zalando Postgres operator. - [ClickHouse Cluster](https://nrp.ai/documentation/userdocs/running/clickhouse/): ClickHouse Cluster on Kubernetes via the Altinity ClickHouse Operator - [Virtualization- General](https://nrp.ai/documentation/userdocs/running/virtualization-general/): Virtualization - General - [Virtualization - Ubuntu](https://nrp.ai/documentation/userdocs/running/virtualization-ubuntu/): Boot an Ubuntu virtual machine on KubeVirt with cloud-init, an SSH key, and scratch storage. - [Virtualization - Windows](https://nrp.ai/documentation/userdocs/running/virtualization-windows/): Install Windows 11 in a KubeVirt virtual machine from an ISO, including the image conversion steps. - [Virtualization - Talos Linux](https://nrp.ai/documentation/userdocs/running/virtualization-talos/): Boot a Talos Linux virtual machine on KubeVirt and reach its API through a LoadBalancer service. - [Ray Cluster](https://nrp.ai/documentation/userdocs/running/ray-cluster/): Run a Ray cluster across nodes with KubeRay for parallel Python and machine-learning workloads. - [Dask Cluster](https://nrp.ai/documentation/userdocs/running/dask-cluster/): Launch a Dask cluster with the Dask operator and scale workers across Nautilus nodes. - [Kubeflow Training](https://nrp.ai/documentation/userdocs/running/kubeflow/): Run distributed training with the Kubeflow Training Operator: setup, roles, and example jobs. - [High I/O Jobs](https://nrp.ai/documentation/userdocs/running/io-jobs/): Structural tricks for high I/O jobs: parallel streams, archive packing, and local scratch on top of Ceph. - [CPU Throttling](https://nrp.ai/documentation/userdocs/running/cpu-throttling/): Why CPU limits throttle workloads through cgroups, and how to set requests and limits to avoid it. - [JupyterHub Service](https://nrp.ai/documentation/userdocs/jupyter/jupyterhub-service/): Sign in to the hosted JupyterHub service, pick an environment, and run notebooks without managing Kubernetes. - [ML/Jupyter pod](https://nrp.ai/documentation/userdocs/jupyter/jupyter-pod/): Run JupyterLab as a single dedicated pod when the shared JupyterHub service does not fit. - [Deploy JupyterHub](https://nrp.ai/documentation/userdocs/jupyter/jupyterhub/): Deploy your own JupyterHub for a class or group: shared resources, standardized images, and access control. - [Deploy JupyterHub Helm Chart Values](https://nrp.ai/documentation/userdocs/jupyter/values/): Example Helm values for a private JupyterHub: CILogon OAuth configuration and notebook image settings. - [Using Coder](https://nrp.ai/documentation/userdocs/coder/coder/): Use Coder, the browser-based IDE on Nautilus, and update older workspaces without losing data. - [Deploying Coder](https://nrp.ai/documentation/userdocs/coder/deploy/): Deploy your own Coder instance in a namespace, with admin control over templates and users. - [NRP-Managed LLMs](https://nrp.ai/documentation/userdocs/ai/llm-managed/): Hosted open-weights LLMs on the National Research Platform - [Available Models](https://nrp.ai/documentation/userdocs/ai/llm-managed/models/): NRP-managed LLM model catalog and feature matrix - [Chat Interfaces](https://nrp.ai/documentation/userdocs/ai/llm-managed/chat-interfaces/): Web and desktop chat interfaces for NRP-managed LLMs - [API Access](https://nrp.ai/documentation/userdocs/ai/llm-managed/api-access/): API access to NRP-managed LLMs via the Envoy AI Gateway - [Client Configurations](https://nrp.ai/documentation/userdocs/ai/llm-managed/client-configs/): Configuration recipes for CLI coding tools using NRP-managed LLMs - [Fair Use Policy](https://nrp.ai/documentation/userdocs/ai/llm-managed/fair-use/): Fair use limits for NRP-managed LLM models - [Model Lifecycle and Changelog](https://nrp.ai/documentation/userdocs/ai/llm-managed/lifecycle/): How NRP-managed LLM models are added, deprecated, and removed - [NRP Accounting MCP Server](https://nrp.ai/documentation/userdocs/ai/accounting-mcp/): Query NRP accounting data over MCP or an OpenAPI bridge. - [LLM in JupyterHub](https://nrp.ai/documentation/userdocs/ai/llm-jupyterhub/): Work with LLMs inside a JupyterHub notebook, hosted or self-run, and size your volume for model weights. - [Cloud AI 100](https://nrp.ai/documentation/userdocs/ai/qaic/): Qualcomm Cloud AI 100 Ultra Cards - [NRP-Managed vector database](https://nrp.ai/documentation/userdocs/ai/vector-database/): Connect to the NRP-managed Milvus vector database from your application: endpoints, keys, and collections. - [LoadBalancer VIPs](https://nrp.ai/documentation/userdocs/networks/loadbalancer-vips/): Use MetalLB LoadBalancer VIPs for IPv4, IPv6, and dual-stack services on Nautilus. - [Joint Experiments with FABRIC Testbed](https://nrp.ai/documentation/userdocs/networks/fabric/): A tutorial on cross-testbed experimentation and how to establish a connection between a FABRIC slice and a Kubernetes pod on Nautilus. - [Requesting FPGAs and JTAG from a Pod](https://nrp.ai/documentation/userdocs/fpgas/using-fpgas-from-pods/): How to write a Pod spec that requests an Alveo U55C card and its USB JTAG cable, and how to identify the right pair at runtime. - [AMD/Xilinx FPGAs](https://nrp.ai/documentation/userdocs/fpgas/vivado-vitis/): Program an AMD/Xilinx U55C FPGA with Vivado and Vitis using the Coder U55C workflow template. - [ESnet SmartNIC](https://nrp.ai/documentation/userdocs/fpgas/esnet/): Use Xilinx Alveo FPGAs as high-performance SmartNICs with the ESnet framework on Nautilus. - [ESnet SmartNIC FPGA - Development](https://nrp.ai/documentation/userdocs/fpgas/esnet_development/): ESnet SmartNIC FPAG Tutorial - Notebook 1/3 - [ESnet SmartNIC FPGA - Building](https://nrp.ai/documentation/userdocs/fpgas/esnet_building/): ESnet SmartNIC FPAG Tutorial - Notebook 2/3 - [ESnet SmartNIC FPGA - Running](https://nrp.ai/documentation/userdocs/fpgas/esnet_running/): ESnet SmartNIC FPAG Tutorial - Notebook 3/3 - [Introduction](https://nrp.ai/documentation/userdocs/storage/intro/): An overview of NRP storage: POSIX volumes and PVCs, choosing a filesystem and region, expanding volumes, and the purging policy. - [Ceph FS / RBD](https://nrp.ai/documentation/userdocs/storage/ceph/): Mount and share CephFS home storage well: the rules that keep the shared filesystem usable for everyone. - [Ceph S3](https://nrp.ai/documentation/userdocs/storage/ceph-s3/): Create buckets on the platform Ceph storage and read or write them from any S3 client. - [CVMFS](https://nrp.ai/documentation/userdocs/storage/cvmfs/): Mount ready-made scientific software and datasets distributed over CVMFS, including the NRP origins. - [Local Scratch](https://nrp.ai/documentation/userdocs/storage/local/): Request node-local NVMe scratch storage when your workload needs I/O faster than the shared filesystem. - [Linstor](https://nrp.ai/documentation/userdocs/storage/linstor/): Linstor storage - [Nextcloud](https://nrp.ai/documentation/userdocs/storage/nextcloud/): Stage, sync, and share files with the platform Nextcloud instance backed by CephFS. - [SyncThing](https://nrp.ai/documentation/userdocs/storage/syncthing/): Continuously sync large file collections between your machine and the platform with SyncThing. - [Mounting FUSE filesystems](https://nrp.ai/documentation/userdocs/storage/fuse/): Mount an S3 bucket or other FUSE filesystem inside a pod using the smarter-devices/fuse resource and the SYS_ADMIN capability. - [Moving Data](https://nrp.ai/documentation/userdocs/storage/move-data/): Choose how to move data into and out of Nautilus by transfer size, file count, and endpoint. - [Purging](https://nrp.ai/documentation/userdocs/storage/purging/): Pruging - [Managing JWT Credentials](https://nrp.ai/documentation/userdocs/storage/jwt-credential/): How to manage JWT credentials on the Nautilus cluster - [Building in Gitlab](https://nrp.ai/documentation/userdocs/development/gitlab/): Build container images with CI pipelines on the platform GitLab instance. - [Private Repos](https://nrp.ai/documentation/userdocs/development/private-repos/): Let pods pull container images from the private GitLab registry in your group or project. - [K8s Gitlab Integration](https://nrp.ai/documentation/userdocs/development/k8s-integration/): Give GitLab CI deploy access to your namespace on Nautilus with the GitLab agent for Kubernetes or the older certificate-based connection. - [Networking](https://nrp.ai/documentation/admindocs/participating/network/): The Science DMZ, jumbo-frame, and firewall requirements for a contributed node, and the ports each service needs reachable. - [Contributing a node](https://nrp.ai/documentation/admindocs/participating/new-contributor-guide/): The procedure for adding a server to Nautilus — what your site and the NRP each run, the network it needs, and the details to email us. - [Connect a Slurm Site with InterLink](https://nrp.ai/documentation/admindocs/participating/interlink/): Connect a participating Slurm site to NRP with InterLink - [Install an InterLink Virtual Node](https://nrp.ai/documentation/admindocs/participating/interlink-kubernetes/): Install an InterLink virtual node for a participating Slurm site - [Ceph S3 for InterLink Jobs](https://nrp.ai/documentation/admindocs/participating/interlink-storage/): Move data between NRP and remote Slurm jobs with Ceph S3 - [Install](https://nrp.ai/documentation/admindocs/perfsonar/install/): Installing a perfSONAR testpoint so a site joins the mesh-orchestrated tests and registers results to the measurement archive. - [MaDDash Links](https://nrp.ai/documentation/admindocs/perfsonar/maddash/): MaDDash dashboard links for checking the status of perfSONAR and GridFTP end-to-end tests. - [A2A Maddash](https://nrp.ai/documentation/admindocs/perfsonar/a2a/): How the all-to-all MaDDash reports connectivity between sites, and how to find the node each zone contributes to the mesh. - [Builds](https://nrp.ai/documentation/admindocs/fiona/builds/): Component specs and pricing for FIONA appliance builds, with links to the per-generation build sheets. - [Install](https://nrp.ai/documentation/admindocs/fiona/install/): How NRP admins install FIONA nodes that join without giving the platform IPMI access. - [Fixing node issues](https://nrp.ai/documentation/admindocs/fiona/fix/): How an admin taints a faulty FIONA node, repairs its containerd certificate config, and returns it to service. - [Removing a node](https://nrp.ai/documentation/admindocs/fiona/remove/): Removing a node from the cluster - [Topology](https://nrp.ai/documentation/admindocs/nrp/topology/): NRP Topology - [Managing AI Models](https://nrp.ai/documentation/admindocs/ai/managing_models/): How to deploy vLLM/SGLang models and manage the Envoy AI Gateway - [Broken Drives](https://nrp.ai/documentation/admindocs/storage/broken-drives/): How an admin drains and removes the Ceph OSDs behind failed drives without forcing a full cluster rebalance. - [Ceph S3](https://nrp.ai/documentation/admindocs/storage/ceph-s3/): Administering Ceph S3 from the Rook tools pod: adding users, deleting buckets, resharding, and finding orphaned objects. - [Ceph Recover](https://nrp.ai/documentation/admindocs/storage/ceph/): Ceph cluster recovery steps run through the kubectl-rook-ceph plugin. - [Ceph Performance](https://nrp.ai/documentation/admindocs/storage/ceph-performance/): Admin notes on tuning Ceph performance for the mix of PLP NVMe and spinning drives common at NRP sites. - [Linstor Broken Drives](https://nrp.ai/documentation/admindocs/storage/linstor/): How to list, detach, and re-attach faulty Linstor storage resources after a drive fails. - [Volume Mounting Troubleshooting](https://nrp.ai/documentation/admindocs/storage/volume-mounting/): How to unstick pods stuck in ContainerCreating when a networking fault prevents volume mounts. - [Origin SyncThing](https://nrp.ai/documentation/admindocs/storage/origin-sync/): How an admin connects the origin datanode SyncThing instance to a user device for large data loads. - [User PVC Issues](https://nrp.ai/documentation/admindocs/storage/user-pvc-issues/): Fixes for common problems that users face with PVCs - [Getting Certs](https://nrp.ai/documentation/admindocs/vault/cert/): The cert-manager Issuer and Certificate definitions services need to pull credentials from Vault. - [Updating IPMI info](https://nrp.ai/documentation/admindocs/vault/ipmi/): How to update the node IPMI credential file stored in Vault and keep the Ansible inventory in sync. - [Hardware](https://nrp.ai/documentation/admindocs/links/hardware/): External references for NRP-tested hardware designs, including the Flash.io Network Appliance. - [Knowledge Base](https://nrp.ai/documentation/admindocs/links/knowledge-base/): Where to find NRP community knowledge: networking and cyberinfrastructure resources, and where to reach the NRP community. - [Software](https://nrp.ai/documentation/admindocs/links/software/): Curated links to data-transfer and network-test software used on the platform, such as GridFTP and ESnet client tools. - [NRP Visualization Tools](https://nrp.ai/documentation/admindocs/links/vis/): Direct links to NRP visualization and telemetry tools, including Grafana and the cluster traceroute service. - [Cluster User Management](https://nrp.ai/documentation/admindocs/cluster/cluster-user-mgmt/): Managing users in the cluster. - [Node Management](https://nrp.ai/documentation/admindocs/cluster/node-mgmt/): Guide for managing and rebooting nodes in the cluster. - [Namespace Management](https://nrp.ai/documentation/admindocs/cluster/ns-mgmt/): Managing namespaces inlcuding creating and deleting namespaces in the cluster. - [Authentik](https://nrp.ai/documentation/admindocs/cluster/authentik/): Authentik configuration and management. - [Etcd purging](https://nrp.ai/documentation/admindocs/cluster/etcd/): Purging etcd database when it exceeds the limit. - [JupyterLab Admin](https://nrp.ai/documentation/admindocs/cluster/jupyterlab-admin/): JupyterLab administration guide. - [KubeVirt GPU](https://nrp.ai/documentation/admindocs/cluster/kubevirt-gpu/): KubeVirt GPU configuration and management. - [GitLab Runners](https://nrp.ai/documentation/admindocs/cluster/gitlab/): A Cluster Admin Guide Running Gitlab CI/CD Runners - [FPGA Administration](https://nrp.ai/documentation/admindocs/cluster/fpga/): Cluster Admin Guide to managing FPGA-equipped nodes, JTAG cables, the device plugins, and shell flashing. - [CDN Services](https://nrp.ai/documentation/admindocs/cluster/cdn-svc/): CDN services configuration and management. - [KubeRay Operator Installation and Management](https://nrp.ai/documentation/admindocs/cluster/ray-operator/): Instructions for installing and managing the KubeRay operator in the Kubernetes cluster. - [SENSE/Multus](https://nrp.ai/documentation/admindocs/cluster/sense-multus/): SENSE/Multus configuration and management. - [MIG GPUs](https://nrp.ai/documentation/admindocs/cluster/mig-gpus/): Managing MIG GPUs in the cluster. - [Security Tools](https://nrp.ai/documentation/admindocs/cluster/security-tools/): Security tools and practices for the cluster. - [CVE Mitigations](https://nrp.ai/documentation/admindocs/cluster/cve-mitigations/): This document is the guideline to mitigate CVEs that do not have a released kernel patch. - [Upgrades Introduction](https://nrp.ai/documentation/admindocs/upgrades/intro/): Introduction to cluster upgrades - [Calico](https://nrp.ai/documentation/admindocs/upgrades/calico/): Upgrade Calico, the cluster networking plugin, following the official manifest-based procedure. - [Cert-manager](https://nrp.ai/documentation/admindocs/upgrades/cert-manager/): Upgrade cert-manager on Nautilus using its static manifest upgrade path. - [Coder](https://nrp.ai/documentation/admindocs/upgrades/coder/): Upgrade the hosted Coder deployment via its Helm chart. - [Elasticsearch](https://nrp.ai/documentation/admindocs/upgrades/elasticsearch/): Upgrade Elasticsearch safely with the Elastic Cloud on Kubernetes operator. - [GitLab](https://nrp.ai/documentation/admindocs/upgrades/gitlab/): Run and upgrade the platform GitLab instance: its settings, upgrades, registry, backups and GitLab agents. - [JupyterHub](https://nrp.ai/documentation/admindocs/upgrades/jupyterhub/): Keep the hosted JupyterHub current: update image versions and the West profiles list. - [Kubernetes Upgrades](https://nrp.ai/documentation/admindocs/upgrades/k8s/): Kubernetes upgrade procedures - [KubeVirt](https://nrp.ai/documentation/admindocs/upgrades/kubevirt/): Upgrade KubeVirt, the add-on that runs virtual machines on the cluster. - [Nextcloud / LanguageTool](https://nrp.ai/documentation/admindocs/upgrades/nextcloud/): Upgrade procedure for Nextcloud and LanguageTool, including major-release sequencing and the NGINX ConfigMap. - [Overleaf](https://nrp.ai/documentation/admindocs/upgrades/overleaf/): Upgrading Overleaf by bumping the image version in GitLab CI and updating the Kubernetes Deployment by hand. - [Nodes](https://nrp.ai/documentation/admindocs/upgrades/nodes/): Upgrade Kubernetes, CRIO, and OS packages on individual nodes with the Ansible upgrade playbooks. - [Prometheus and Grafana](https://nrp.ai/documentation/admindocs/upgrades/prometheus/): Deploy upgrades for Prometheus, Grafana, and the dashboards generated by the monitoring pipeline. - [Postgres operators](https://nrp.ai/documentation/admindocs/upgrades/postgres-operator/): Zalando and CloudNativePG operators - [ClickHouse operator](https://nrp.ai/documentation/admindocs/upgrades/clickhouse-operator/): Altinity ClickHouse operator - [Rook/Ceph Upgrades](https://nrp.ai/documentation/admindocs/upgrades/rook/): Rook and Ceph upgrade procedures - [Linstor](https://nrp.ai/documentation/admindocs/upgrades/piraeus-operator/): Upgrade the Piraeus/Linstor operator, including the local manifests and patched resource limits. External material linked from the sidebar: - [MNIST Training (Presentation)](https://docs.google.com/presentation/d/1GMvaZr9Nm6LhYUU_E0E0LdoebPpk0dgb2Z6oS9v2Ww8/edit?usp=sharing) (User Guide > Tutorials) ## Documentation --- # Getting Started with the National Research Platform - URL: https://nrp.ai/documentation/ - Description: Orientation to the NRP: how to get access, what you can run, and where to find the services the National Research Platform provides. The **National Research Platform (NRP)** is an NSF-funded computing platform spanning more than 70 research sites. **Nautilus** is its main Kubernetes cluster, and most of these pages describe it: you get access, work inside a project namespace, then run GPU pods, batch jobs, or notebooks on it. - [Getting access](https://nrp.ai/documentation/userdocs/start/getting-started/): Log in through CILogon, join a namespace, and install the tools. - [Tutorials](https://nrp.ai/documentation/userdocs/tutorial/introduction/): A guided path through containers, pods, jobs, and storage. - [Cluster Policies](https://nrp.ai/documentation/userdocs/start/policies/): Acceptable use and access policies for the platform. - [FAQ](https://nrp.ai/documentation/userdocs/start/faq/): Quick answers to the questions we get most often. ## Your first job, step by step 1. [Getting access](https://nrp.ai/documentation/userdocs/start/getting-started/) — sign in through CILogon and install `kubectl`. 2. Join a project. Work runs inside a namespace your [Group Administrator](https://nrp.ai/documentation/userdocs/start/hierarchy/) creates; getting-started shows how to request one if you do not yet have it. 3. Start with a [GPU pod](https://nrp.ai/documentation/userdocs/running/gpu-pods/) for interactive work, or a [batch job](https://nrp.ai/documentation/userdocs/running/jobs/) for runs that finish on their own. 4. Keep your results on [persistent storage](https://nrp.ai/documentation/userdocs/storage/ceph/) and [move data in and out](https://nrp.ai/documentation/userdocs/storage/move-data/) when you are done. :::caution[Data policy] The National Research Platform currently has no storage that is suitable for **HIPAA**, **PID**, **FISMA**, **FERPA**, or protected data of any kind. Users are not permitted to store such data on NRP machines. Unsure whether your data counts as protected? [Ask us](https://nrp.ai/documentation/userdocs/start/support/) before you store it. ::: ## Beyond your first job Everything linked here complements the steps above — not repeats of them. **Ways of working** - [Best ways of using the NRP](https://nrp.ai/documentation/userdocs/start/using-nautilus/) — recommended patterns for everyday use. - [Run CPU-only jobs](https://nrp.ai/documentation/userdocs/running/cpu-only/) — batch work that does not need GPUs. - [Run a notebook](https://nrp.ai/documentation/userdocs/jupyter/jupyter-pod/) — Jupyter inside a pod on your own resources. **Apps and data** - [Expose an app over HTTP](https://nrp.ai/documentation/userdocs/running/ingress/) — give your service a public URL. - [Find a base image](https://nrp.ai/documentation/userdocs/running/sci-img/) — scientific images the NRP maintains. - [Use the managed LLM endpoints](https://nrp.ai/documentation/userdocs/ai/llm-managed/) — models the NRP hosts for you. - [Deployed services](https://nrp.ai/documentation/userdocs/start/resources/) — data sources and tools; many require separate registration. ## Live cluster status Where the cluster stands right now: - [Cluster resources](https://nrp.ai/viz/resources/): Current availability across the platform's clusters. - [NRP site map](https://dash.nrp-nautilus.io/map): Every NRP site and its current state. ## For AI assistants [`llms.txt`](https://nrp.ai/llms.txt) is these docs as one plain-text Markdown file, led by the rules an AI assistant must follow when helping with Nautilus. Give it to your assistant, or point the assistant at `https://nrp.ai/llms.txt`. --- # Getting access - URL: https://nrp.ai/documentation/userdocs/start/getting-started/ - Section: User Guide > Start - Description: Log in through CILogon, get added to a namespace, and install the tools you need to reach the Nautilus cluster. ## Get access and log in If you are a new user and want to access the Nautilus cluster, follow the steps below. :::note[The namespaces manager has moved] The namespaces management page is now at [https://nrp.ai/namespaces](https://nrp.ai/namespaces). If you can't find the namespace in the old portal, try the new one. ::: 1. Point your browser to the [NRP portal](https://nrp.ai). 2. On the portal page, click on the **`Log In`** button at the top right corner. 3. You will be redirected to **`CILogon`** — the federated login service shared by research institutions — where you have to **Select an Identity Provider**. 4. Select your institution (for example: University Name) from the menu and click the **`Log On`** button. - If your institution is using **`CILogon`** as a federated certification authority, it will be on the menu. Select the name of your institution and use either a personal account or an institutional G-suite account. This is usually your institutional account name (email) and a password associated with it. - If your institution is not in the list, you may be able to use the **`Microsoft`** entry. If your university uses Microsoft's Active Directory for user management (common if your campus uses Office 365 or related products), then you can login with your institutional credentials at the Microsoft login prompt. - If your institution is not in the **`CILogon`** dropdown, and Microsoft also doesn't work, you can select **`Google`** or **`Github`**. :::caution[Note for ORCID users] We require email to be visible. If you are using **`ORCID`** as your institution, please change the email setting to make it visible to everyone. ::: 5. After a successful authentication, you will log in to the portal and your account will be created if it was not created before. 6. If you are a **student**, please contact your research supervisor and ask them to add you to their [namespace](https://nrp.ai/documentation/userdocs/start/using-nautilus/#namespace). Once you are added to a namespace, your status will change to a cluster **`user`** and you will get access to all namespace resources. 7. If you are a **faculty member, university staff member, or postdoc** starting a new project and need your own [namespace](https://nrp.ai/documentation/userdocs/start/using-nautilus/#namespace) — either for yourself or your research group — request one by filling out the [namespace request form](https://get-started.nrp-nautilus.io). Log in at [nrp.ai](https://nrp.ai) first so the request is tied to your account; students should have their advisor submit the form. The form asks for the desired name for your group (usually the department name) — use lowercase letters, numbers, and dashes only, and choose clear, descriptive names (avoid generic ones like `kubernetes-ai`, `testing-group`, `llm-access`). To see the existing names for reference, visit this [visualization tree](https://nrp.ai/viz/namespaces/). If you have questions, ask in [Nautilus Support](https://nrp.ai/contact). As an **`admin`**, you will have the ability to: - Create multiple namespaces. - Invite other users to your namespace(s). :::note[Admin Responsibilities] As a namespace **`admin`**, you will be the one responsible for - all activity happening in your namespaces, - keeping the user list up-to-date. ::: We're handling the namespaces management as a [hierarchical model](https://nrp.ai/viz/namespaces) where we delegate the management of certain parts to the admins who can then further delegate the responsibilities.. The form asks for your current affiliation; it helps to think about where you see yourself fitting in this structure. You can simply own a single namespace under your university branch for small experiments or manage multiple different namespaces for large departments or groups of students. [Read more](https://nrp.ai/documentation/userdocs/start/hierarchy) 1. Access your namespace(s). - If you have become a namespace **`admin`**, you can start creating your own namespaces at this time: open the [**Namespaces**](https://nrp.ai/namespaces) page, select your namespace, and use its **Subgroups** tab. Add other users on its **Members** tab. People who haven't logged in to the portal yet can be invited by email, and are added when they first log in. - If you got added to a namespace as a **`user`**, you can verify it on the [**Namespaces**](https://nrp.ai/namespaces) page after logging in to the portal. The namespaces you're a member of are listed under **Mine**. 2. **Make sure you read the cluster policies before starting to use it.** :::caution[Read these before you run anything] - Read the [**NRP Acceptable Use Policy (AUP)**](https://nrp.ai/NRP-AUP.pdf). - Read the [**Cluster Policies**](https://nrp.ai/documentation/userdocs/start/policies/). ::: 3. **Make sure you're on Nautilus Support. All Nautilus and NRP users are expected to join for user support and real-time updates.** :::note[Join the Nautilus Support chat] - If you do not yet have access to Nautilus Support, please register right away following the instructions at [**this link.**](https://nrp.ai/contact/) ::: ## Cluster access via kubectl :::note[Do not skip any steps] - Please do not skip any steps. **All steps below are necessary to be able to connect to Nautilus.** - Common Kubelogin connectivity issues are addressed in Step 3. Please view carefully if you run into any issues. ::: 1. [Install][1] the Kubernetes command-line tool, `kubectl` from the [official instructions][1] of each OS. 2. [Install kubelogin](https://github.com/int128/kubelogin?tab=readme-ov-file#setup) plugin :::caution[Install kubelogin first] You **must** install the `kubelogin` plugin, or your kubeconfig file **will not work**. ::: [Download kubelogin](https://github.com/int128/kubelogin?tab=readme-ov-file#setup) Example for Linux or macOS (`curl`, `jq`, and `unzip` should be installed): ```bash # OS_ARCHITECTURE should be in the format `amd64` or `arm64` # For Debian/Ubuntu - change as adequate for RHEL or macOS OS_ARCHITECTURE="$(dpkg --print-architecture)" # Change to `darwin` for macOS OS_NAME="linux" # Install kubelogin KUBELOGIN_VERSION="$(curl -fsSL "https://api.github.com/repos/int128/kubelogin/releases/latest" | jq -r '.tag_name')" curl -o kubelogin.zip -fSL "https://github.com/int128/kubelogin/releases/download/${KUBELOGIN_VERSION}/kubelogin_${OS_NAME}_${OS_ARCHITECTURE}.zip" unzip kubelogin.zip kubelogin chmod +x ./kubelogin # Replace `/usr/local/bin` to `~/.local/bin` (create the directory when nonexistent) and set `export PATH="~/.local/bin:${PATH}"` for local user sudo mv ./kubelogin /usr/local/bin/kubectl-oidc_login sudo chown root: /usr/local/bin/kubectl-oidc_login rm -f kubelogin.zip ``` 2. Save the [config file](https://nrp.ai/config) as **`config`** (without any extension) and put it in your `$HOME/.kube` folder. [Download Config File](https://nrp.ai/config) This folder may not exist on your machine, to create it execute: ```bash mkdir ~/.kube ``` Finally, you should have the Nautilus Kubernetes config on your machine (laptop, desktop) as `~/.kube/config` (e.g., `curl -o ~/.kube/config -fSL "https://nrp.ai/config"`) 3. **Cross-platform kubelogin fixes (WSL / Linux / Windows / macOS)** - **No local browser available** (remote console, use the link in `Please visit the following URL in your browser:` to login with CILogon): ```bash --grant-type=device-code --skip-open-browser ``` - **Browser issues** (won’t launch or opens incorrectly): ```bash --browser-command="/mnt/c/Program Files/Google/Chrome/Application/chrome.exe" ``` (Windows WSL: points to Windows Chrome) - **Port binding errors** (`port 8000 already in use`): ```bash --listen-address=0.0.0.0:18000 ``` (change to any unused port) **Example config snippet:** ```yaml args: - oidc-login - get-token - --browser-command="/mnt/c/Program Files/Google/Chrome/Application/chrome.exe" - --listen-address=0.0.0.0:18000 - --token-cache-storage=disk ``` You can store the token cache to the OS keyring for enhanced security. ```yaml args: - oidc-login - get-token - --token-cache-storage=keyring ``` 4. Make sure you are using the correct config file. - Run the following command to list available Kubernetes contexts: ```bash title="Get Contexts" $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * nautilus nautilus oidc ``` - If you have access to multiple Kubernetes clusters, you need to choose `use-context nautilus` by doing ```bash title="Set Context" kubectl config use-context nautilus ``` 5. Issue the kubectl command. It will authenticate via CiLogon by opening the browser window, and close one in the end. ``` kubectl get nodes ``` 6. Verify cluster access using `kubectl`. Run the following command on your terminal. ```bash title="Get Pods" kubectl get pods -n ``` If you see the message "No resources found in your namespace" it means there are no pods in your namespace yet. This indicates that you have access to the resources of your namespace. 7. If you know you're a member of a namespace, you can set it as default. ``` kubectl config set contexts.nautilus.namespace ``` ## Updating namespace membership By default the access token expires in half an hour and is automatically updated after that. If you need to update one sooner (for example, you were added to a new namespace), you can invalidate the token: ```bash title="Invalidate access token" kubectl oidc-login clean ``` After that any `kubectl` command will trigger getting the new token. (F.e. `kubectl get nodes`) :::danger[Read this before you run anything on the cluster] - Containers are stateless. - **All your data will be gone forever when the container restarts**, unless you store it in a persistent volume. - Container restart is normal in the Kubernetes (k8s) cluster. Expect it. - Never force Delete Pods. - Users running a **`Job`** with **`sleep` command** or equivalent (any command that never ends by itself) will be banned from using the cluster. ::: ## Use the Cluster 1. Read the [Using Nautilus](https://nrp.ai/documentation/userdocs/start/using-nautilus) page to learn how to use the cluster. 2. To learn more about Kubernetes you can look at [our tutorial](https://nrp.ai/documentation/userdocs/tutorial/basic/). 3. Other helpful resources: * [`kubectl` Tool Overview][4] and [Cheatsheet][6] * [Kubernetes Basics][3] from Kubernetes project * [Kubernetes Official Tutorials][5] Please note that not all examples will work in our cluster because of security policies. You are limited to seeing what's happening in your own namespace, and nobody else can see your running pods. ## GUI tools for Kubernetes You might want to try one of these GUI tools for Kubernetes: * [K9s](https://k9scli.io/) - Console graphical user interface * [Visual Studio Code](https://code.visualstudio.com/) - **Instructions below** * [Lens](https://k8slens.dev/) - Graphical user interface They will all use your `config` file in the default location to get access to the cluster. ### Visual Studio Code
Open Instructions Install both the [Kubernetes](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) and [Remote Development](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) extensions. After configuring the requirements below and setting the namespace (look at the bottom of the interface after entering the Kubernetes tab), right-click the Deployment, Pod, or any other resource, and press **`Attach Visual Studio`**. **Two things are required to work:** - The Nautilus cluster `config` file must be located precisely in `~/.kube/config` (Linux, macOS), or `%USERPROFILE%\.kube\config` (Windows). Using `--kubeconfig` or manually specifying the kubeconfig file path in Visual Studio Code will not work. - The interface prompt for installing `kubectl` within Visual Studio Code will **NOT** work for accessing the resource. **Both `kubectl` and `kubectl-oidc_login` executable files** must be in the `PATH`. - For **Windows**, an easy way to make `kubectl` and `kubectl-oidc_login` work is to copy the two `.exe` executable files into `%LOCALAPPDATA%\Programs\Microsoft VS Code\bin` or `C:\Program Files\Microsoft VS Code\bin` (depending on the Visual Studio Code installation), where these directories are already added into the `PATH` in Windows.
[1]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ [2]: https://nrp.ai [3]: https://kubernetes.io/docs/tutorials/kubernetes-basics/ [4]: https://kubernetes.io/docs/reference/kubectl [5]: https://kubernetes.io/docs/tutorials/ [6]: https://kubernetes.io/docs/reference/kubectl/cheatsheet/ --- # Using Nautilus - URL: https://nrp.ai/documentation/userdocs/start/using-nautilus/ - Section: User Guide > Start - Description: guide to use the cluster using kubernetes, jupyterhub and coder Nautilus is a Kubernetes cluster distributed all over the US and there are a few sites in Asia and Europe. Currently there are over 75 sites as part of the NRP. See all the sites on the [NRP dashboard](https://dash.nrp-nautilus.io/). :::note[Cluster Resource Summary] The cluster has different types of resources including CPU, GPU,and FPGAs. - **CPUs:** 16 - 384 cores / node - **GPUs:** max 16 in a node - **FPGAs:** max 4 in a node - **Memory (RAM):** 16 GB - 1.6 TB / node - **Disk Space (local scratch):** 107.3 GB - 17.8 TB / node Check the [**Resources**](https://nrp.ai/viz/resources) page on the portal to see the detailed summary of each node. - The page also shows the available resouces in real time. Check the cluster summary on [**Grafana**](https://grafana.nrp-nautilus.io/d/KQsJTWPiz/cluster-summary-nrp?orgId=1&from=now-1h&to=now&timezone=browser) ::: ## How to use the cluster There are three main ways to use the computing resources on the Nautilus cluster. 1. NRP-hosted [JupyterHub platform](https://jupyterhub-west.nrp-nautilus.io) 2. NRP-hoster [Coder platform](https://coder.nrp-nautilus.io) 3. Interfacing with Kubernetes using the `kubectl` tool 4. Using the [managed cluster services](https://nrp.ai/documentation/userdocs/start/resources) We will discuss these options below. ### JupyterHub Platform [JupyterHub](https://nrp.ai/documentation/userdocs/jupyter/jupyterhub-service) is arguably the most user-friendly way to interact with the Nautilus cluster. It allows you to run [Jupyter notebooks](https://jupyter.org/) in a web browser, without having to worry about the underlying infrastructure. You can access the NRP-hosted JupyterHub platform by visiting the [JupyterHub](https://jupyterhub-west.nrp-nautilus.io) link and logging in with your institutional credentials. Once authenticated, you can choose the hardware specs to spawn your instance and run Jupyter notebooks as usual.
[Access the NRP-hosted JupyterHub](https://jupyterhub-west.nrp-nautilus.io)
:::tip[Points to Note] - You **do** need to be part of a namespace to use the NRP-hosted JupyterHub service. - Your institution needs to be part of **`CILogon`**. - Look at the [**JupyterHub documentation**](https://nrp.ai/documentation/userdocs/jupyter/jupyterhub-service) to learn more about this option. ::: ### Coder Platform [Coder](https://nrp.ai/documentation/userdocs/coder/coder) provides an easy-to-use, JupyterHub-like experience, and you can use the Nautilus cluster without Kubernetes knowledge. [Coder](https://coder.com/) also runs on the web browser. You can run your code without worrying about the underlying infrastructure. You can access the NRP-hosted Coder platform by visiting the [Coder](https://coder.nrp-nautilus.io) link and logging in with your institutional credentials using **OpenID Connect** (once the cluster admins approve your account).
[Access the NRP-hosted Coder](https://coder.nrp-nautilus.io/)
:::tip[Points to Note] - You **do** need to be part of a namespace to use the NRP-hosted Coder service. - Cluster admins need to approve your account. Ask for access in [Nautilus Support](https://nrp.ai/contact). - Your institution needs to be part of **`CILogon`**. - Look at the [**Coder documentation**](https://nrp.ai/documentation/userdocs/coder/coder) to learn more about this option. ::: ### Kubernetes This method provides greater control over your computing resources but requires basic Kubernetes knowledge. You can create pods, jobs, and deployments while specifying the required CPU, GPU, memory, and other resources. This is particularly useful for running custom software stacks or jobs with specific resource requirements.
[Get started with Kubernetes setup](https://nrp.ai/documentation/userdocs/start/getting-started)
:::tip[Points to Note] - You need to be part of at least one namespace. Follow the [Getting Started](https://nrp.ai/documentation/userdocs/start/getting-started) page - Your institution needs to be part of **`CILogon`**. - Need to know basic Kubernetes. - Complete the [**Basic Kubernetes**](https://nrp.ai/documentation/userdocs/tutorial/basic) tutorial. ::: ## Kubernetes concepts This section highlights some of the Kubernetes concepts that are essential to start using the Nautilus cluster. You can learn more about these components in the [tutorial sections](https://nrp.ai/documentation/userdocs/tutorial/basic/). If you are new to Kubernetes and like to learn about the Kubernetes concepts in detail, please follow the [**Concepts page from Kubernetes Documentation**](https://kubernetes.io/docs/concepts/). ### Namespace One of the main concepts of running workloads in Kubernetes is a namespace. A namespace creates an isolated environment in which you can run your pods. It is possible to invite other users to your namespace and it is possible to have access to multiple namespaces. Each namespace has at least one namespace admin (usually researchers, engineers, or faculties). If you are new to the cluster, please follow the instructions on the [Getting Started](https://nrp.ai/documentation/userdocs/start/getting-started#get-access-and-log-in) page. :::tip[Who can be a Namespace Admin] - A **faculty member, researcher, or postdoc**. Request a namespace with the [namespace request form](https://get-started.nrp-nautilus.io) — it asks for your name, institution, and preferred namespace name. Questions can go to [Nautilus Support](https://nrp.ai/contact). - A student (preferably a Ph.D. or Masters student) can be a namespace admin if their supervisor submits the [form](https://get-started.nrp-nautilus.io) on their behalf. ::: ### Container images A container image is a lightweight, standalone, and executable software package containing everything needed to run a piece of software: the code, runtime, libraries, dependencies, and default values for any essential settings. Images are typically stored in container registries (e.g., Docker Hub, GitLab Container Registry) and can be pulled by Kubernetes nodes when running your code. :::tip - Create an image (e.g. Docker image) to run your code on Nautilus. - Store it in one of the container registries. You can use [NRP-hosted GitLab](https://gitlab.nrp-nautilus.io/) to store your container image. ::: ### Pod, Job and Deployment A [Kubernetes pod](https://kubernetes.io/docs/concepts/workloads/pods/) is the smallest and most basic deployable unit in Kubernetes, representing a single instance of a running process. A pod can contain one or more tightly coupled containers that share the same network and storage. If you frequently deploy single-container pods, you can generally replace the word "pod" with "container." :::caution Single pods without any Workload Controllers will be destroyed after **6 hours**. Look at the [**cluster policies**](https://nrp.ai/documentation/userdocs/start/policies#interactive-use-6-hours-max-runtime). - Pods are ephemeral and get deleted or evicted due lack of resources on the node or other node issues. - Be careful while using a single pod without any controller. ::: A [Kubernetes job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) is a resource used to run short-lived, batch, or parallel processing tasks to completion. It creates one or more pods and retries execution until the specified number of successful completions is reached. It tracks completed pods and terminates once the specified number of successes is achieved. Deleting a job will clean up all the jobs it created. :::caution - Do not run long `sleep` inside a job. See the Look at the [**cluster policies**](https://nrp.ai/documentation/userdocs/start/policies#batch-jobs) - Do not submit more than 400 Jobs at once. - If you find an issue in your job, delete the job otherwise it will keep creating pods. ::: A [Kubernetes deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) manages the lifecycle of application workloads by ensuring the desired number of pod replicas are running and up to date. It supports rolling updates, rollbacks, and scaling to maintain application availability. Deployments automatically replace failed or outdated pods, making them ideal for managing stateless applications in a cluster. :::caution - Deployments are automatically deleted after 2 weeks unless you ask for an exception. - Use deployments if you need to run [long-running idle pods](https://nrp.ai/documentation/userdocs/running/long-idle/). ::: :::note In almost every case, **Kubernetes jobs are the preferred way to run large-scale processing on the NRP**. This is because jobs are designed to run to completion, and can be scaled up or down as needed. ::: | Feature | Job | Pod | Deployment | |---------|-----|-----|------------| | **Description** | A job is a task that runs to completion. | A pod is a group of one or more containers, with shared storage and network resources. | A deployment is a way to manage a collection of pods. | | **Management** | Runs jobs to successful completion, handles node failures | Runs a container, no handling of node failure or conatiner failure | Manages pods, scales up or down, handles conatiner failures | | **Paralellism** | Can run arrays of jobs, or parallel jobs (see [Kubernetes docs](https://kubernetes.io/docs/concepts/workloads/controllers/job/#parallel-jobs)) | Can run multiple containers in a pod | Can run multiple replicas of a pod | | **Max Resources** | Can specify resources needed for the job | 2 GPUs, 32 GB RAM and 16 CPU cores | Can specify resources needed for the deployment | | **Max Runtime** | Runs to completion | 6 hours | 2 weeks | | **Documentation** | [Jobs](https://nrp.ai/documentation/userdocs/running/jobs) | [Pods](https://kubernetes.io/docs/concepts/workloads/pods/) | [Deployments](https://nrp.ai/documentation/userdocs/running/long-idle) | ### Resource allocation For all the abovementioned Kubernetes objects (pods, jobs, deployments, etc) you need to ask for computing resources like (CPU, GPU, memory) before creating the objects. - A **`request`** is the minimum amount of CPU, GPU, or memory that Kubernetes guarantees to a container. Kubernetes scheduler reserves this amount of resources to schedule a pod. - A **`limit`** is the maximum amount of CPU or memory that a container is allowed to use. :::caution - If a node cannot satisfy the resource requests, the pod will not be scheduled on that node. - If a container exceeds the memory limit, it will be killed (OOMKilled - Out of Memory). - If a container exceeds the CPU limit, Kubernetes throttles it (slows down execution). ::: ### Storage A [Kubernetes storage](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) provides a way to persist and manage data for containerized applications. It supports various storage options, including temporary storage (emptyDir), and persistent storage (PersistentVolumes). On Nautilus, you will be using both temporary (`ephemeral-storage`) and persistent storage for your work. A pod can access the requested `ephemeral-storage` from the local disks of the node. Users need to create their own [PersistentVolumeClaim (PVC)](https://nrp.ai/documentation/userdocs/tutorial/storage#creating-a-persistent-volume-claim) under their namespace. > **Important:** On Nautilus, pods that write more than **50Gi** of ephemeral scratch data per container (for example, to an `emptyDir`) can be evicted. If you attach an `emptyDir` scratch volume and expect to write more than 50Gi, set `resources.requests.ephemeral-storage` (and optionally `resources.limits.ephemeral-storage`) in your pod spec to the size you need. See the [Storage tutorial](https://nrp.ai/documentation/userdocs/tutorial/storage) and [Local Scratch](https://nrp.ai/documentation/userdocs/storage/local) pages for complete YAML examples that show both `emptyDir` and `ephemeral-storage` configured together. :::tip - Complete the Storage section from [the tutorial](https://nrp.ai/documentation/userdocs/tutorial/storage). - Read more about the different [storage options](https://nrp.ai/documentation/userdocs/storage/intro) on Nautilus. ::: ### System load While you are welcome to use as many resources as needed, using those inefficiently causes problems for others. The CPU load of the system consists of user load, system load, and several others. If you run `top`, you'll see something like: ```bash %Cpu(s): 26.9 us, 1.5 sy, 0.0 ni, 71.5 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st ``` This means 26.9% of all system CPU time is spent on user tasks (computing), 1.5% on system stuff (kernel tasks), 71.5 idle, and so on. If the system (kernel) load is more than \~15%, it indicates a problem with the user's code. Usually, it is caused by too many threads (the system is spending too much time just switching between those), inefficient file I/O (lots of small files processed by too many workers), or something similar. In this case, the system time is wasted not on processing. > If you were told your pod is causing too much load, look at your code and try to figure out why it is spending too much kernel time instead of computations. --- # How National Research Platform (NRP) Manages Resources - URL: https://nrp.ai/documentation/userdocs/start/hierarchy/ - Section: User Guide > Start - Description: How the NRP organizes users and resources into Organizations, Labs, and Projects, and what that means for the namespaces and services your group can use. The National Research Platform (NRP) is using a novel way for managing computational resources. This new model allows managing the team's access and resources without needing to contact the NRP core team for every change, enabling everyone to get their work done faster and more efficiently. :::note[Membership changes take a moment to reach the cluster] - The portal updates the caches for groups info every minute. - The k8s access token expires in half an hour. If you need to update the group membership sooner, use the [oidc-login clean command](https://nrp.ai/documentation/userdocs/start/getting-started#updating-namespace-membership). ::: ## How it works: the hierarchical system The core of this upgrade is a **hierarchical management system**. This structure organizes users and resources in a clear, nested hierarchy, much like a folder structure: - **Organizations:** The top level, such as a university or a large research consortium. - **Labs:** Groups within an organization, typically led by a faculty member or primary researcher. - **Projects:** Specific research initiatives within a lab. This structure allows designated administrators to manage permissions for their own members across all their labs and projects. ### How this maps to what you use day to day The documentation and the cluster itself use Kubernetes terms in most places. These are the same objects seen from two angles: | In the hierarchy | On the cluster | | ------------------------ | ----------------------------------------------------------------------------------------- | | **Project** | A Kubernetes **namespace** | | **Group Administrator** | The **namespace `admin`** who adds you to a namespace and answers for what runs in it | | **Lab** / **Organization** | Grouping above the namespace — it carries membership and allocation, not a namespace itself | ## Logging in Access to the NRP is managed through **Authentik**, our single-sign-on authentication system. It connects to your existing university or institutional account via [CILogon](https://www.cilogon.org). 1. The first time you log in, you will be required to read and accept the [NRP Acceptable Use Policy (AUP)](https://nrp.ai/NRP-AUP.pdf). 2. Accepting the AUP registers your account. It does **not**, on its own, give you compute — you get that once a Group Administrator adds you to one of their Projects. [Getting access](https://nrp.ai/documentation/userdocs/start/getting-started/) walks through both halves, including how to request administrator rights if you are starting a new group. Some [NRP services and resources](https://nrp.ai/documentation/userdocs/start/resources/) require their own separate registration; each one says so on its own page. ## Key resources and how they're managed This new hierarchical model applies to all resources you use on the platform. ### Kubernetes compute Our primary service is providing compute resources via Kubernetes. Under the new system, each **Project** you create in the hierarchy directly corresponds to a **namespace** in the Kubernetes cluster. This gives you a secure, dedicated space for your team's applications and workflows. ### LLM proxy Groups can have the LLM Proxy capability, allowing creating LLM tokens in those. ### Vector database Groups can have the Vector DB capability, which creates the database in our managed Vector DB. ## Group Administrators Users with **faculty**, **researcher**, or **postdoc** status can request administrator permissions for their Lab or Organization. Admins can: - Add or remove users from their groups. - Create new Projects (i.e., Kubernetes namespaces). - Manage resource allocations for their teams. A Group Administrator is personally responsible for everything that runs in their Projects, and for keeping the member list current. ## Fair share scheduling and resource allocation :::note[Not in effect yet] Scheduling integration is planned, not deployed. Until it ships, jobs are scheduled as described in [Cluster Policies](https://nrp.ai/documentation/userdocs/start/policies/). ::: The resource hierarchy will be integrated with scheduling add-ons to manage how compute jobs are prioritized and allocated. To ensure fairness and reward contributions, groups that donate hardware to the cluster will receive an increased resource allowance. This bonus is automatically passed down to all the labs and projects within that group, ensuring your team directly benefits from your contributions on top of the base allocation available to everyone. If your institution is weighing a hardware contribution, [Contribute hardware](https://nrp.ai/contribute) covers what is involved. --- # Training Join Links - URL: https://nrp.ai/documentation/userdocs/start/training-join-links/ - Section: User Guide > Start - Description: Give workshop attendees time-limited access to your namespace, and optionally an LLM API key, with a single link A training join link lets attendees into your namespace without waiting for you to add them one by one. Anyone who opens the link and logs in to the NRP is added to the namespace. You can also have the link give each attendee an LLM API key. When the training ends, people the link added are removed again. ## What attendees get - **Membership of the namespace**, until access ends. - **Kubernetes `edit` access**, if the namespace has Kubernetes enabled. This is the same access any member of the namespace has. - **An LLM API key**, if you turned on **Also give each attendee an LLM API key**. This option is only offered on namespaces with LLM access. Use a namespace set up for the training if attendees should not see your other work. ## Before you start - You must be an admin of the namespace. - Decide how long access should last. It can last at most 90 days. ## Create a link 1. Open [Namespaces](https://nrp.ai/namespaces), select the namespace, and open its **Join links** tab. 2. Under **Training join links**, enter a training name, the maximum number of attendees, when joins stop being accepted, and when access ends. On namespaces with LLM access, choose whether attendees also get an LLM API key. 3. Select **Create join link**, then copy the link from the table and share it with attendees. ## What attendees see The link opens the [Join a Training](https://nrp.ai/join) page. Attendees log in to the NRP with their institution, or another provider offered on the login page. Their first login creates their NRP account. They are then added to the namespace and told when their access ends. - If the link gives LLM keys, the page shows the attendee's key and the base URL to use. If they lose the key, they can open the link again. That replaces their training key and doesn't use up another seat. - If the namespace has Kubernetes enabled, the page points them to the `kubectl` setup instructions. ## When the training ends - **Revoke** stops new joins. People who already joined keep access until it ends. - **End now** stops new joins and removes attendees within 15 minutes. - At the access end date, everyone the link added is removed from the namespace within 15 minutes, and you get an email summary. Their LLM keys in the namespace are revoked a few minutes later. A `kubectl` login they already had can keep working for up to 30 more minutes, until its token expires. People who were already members of the namespace, people you added yourself, and namespace admins are never removed. Attendees keep their NRP accounts, so their usage stays in NRP reports. --- # Cluster Policies - URL: https://nrp.ai/documentation/userdocs/start/policies/ - Section: User Guide > Start - Description: cluster policies on how to use resources, request gpus, run interactive pods, batch jobs and deployments :::note[TL;DR] - Use [**`Job`**](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) to run [batch jobs](https://nrp.ai/documentation/userdocs/running/jobs/) and set right resources request. - Use [**`Deployment`**](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) if you need a [**long-running pod**](https://nrp.ai/documentation/userdocs/running/long-idle/) and set **minimal** resources request. - Use [monitoring](https://nrp.ai/documentation/userdocs/running/monitoring/) to accurately set the resource requests. - Avoid wasting resources: if you’ve requested something, use it, and free it up once computation is done. Admins monitor resource utilization and will ban namespaces with under-utilized resources requests. - A single user can't submit **more than 4 pods** violating the following limits: - GPU > 40%, CPU 20-200%, RAM 20-150% of requested amount. - Users running a **Job with `sleep` command** or equivalent (script ending with "sleep") will be banned from using the cluster. - Clean up your data frequently. Any storage volume that was **not accessed for 6 months can be purged without notification**. ::: ## LLM Usage :::caution[LLM Usage] Please use the [llms.txt](https://nrp.ai/llms.txt) file when using AI/LLM agents with NRP/Nautilus. Usage of [llms.txt](https://nrp.ai/llms.txt) is likely to prevent most policy violations, as well as providing a better response. ::: ## Acceptable Use Policy Read the [NRP Acceptable Use Policy (AUP)](https://nrp.ai/NRP-AUP.pdf). :::caution[You need to accept the AUP before using the cluster] ::: ## Resource allocation 1. Set the resource (particularly CPU, memory and ephemeral-storage) `limits` properly. Your resource `limits` must be within 20% of the resource `requests`. 3. While running a large number (> ~100) of pods or jobs, you must choose your resource `limit` = `request`. 4. If a pod goes over its memory limit, it will be killed (OOM - Out of Memory). 5. If a pod exceeds the CPU limit, it puts pressure on the node. It can affect the whole node and other use's work. :::tip It is crucial to set `limits` correctly, but it is equally important not to set `requests` too high. - Aim to match your `requests` closely to the average resources you expect to use, while setting `limits` slightly above your highest anticipated peak. - Use [monitoring](https://nrp.ai/documentation/userdocs/running/monitoring/) to fine-tune these values effectively. ::: 6. Do not waste resources. Namespaces with consistently underutilized requests risk being banned. ## Resource usage violations 1. A user can not submit **more than 4 pods**, violating the following conditions: - **GPU** utilization is more than 40% of the requested GPUs. - **CPU** usage is within 20-200% of the requested CPUs. - **Memory (RAM)** usage is within 20-150% of the requested memory. 2. These restrictions **do not apply** to pods requesting **1 CPU core and 2GB memory**. 3. Keep checking the [**Violations**](https://nrp.ai/userinfo) page on the Nautilus portal to see if your pods are violating the usage policies. ## Interactive use (6 hours max runtime) 1. Pods do not stop on their own under normal conditions and won’t recover if a node fails. Because of this, we assume any pod running without a controller is **interactive**, meaning it's used temporarily for development or debugging. 2. **Time limit:** such pods will be **destroyed in 6 hours** unless you request an exception for your namespace (in case you run JupyterHub or some other application controlling the pods for you). 3. **Resource limits:** interactive pods are limited to 2 GPUs, 32 GB RAM, and 16 CPU cores. 4. It is okay to run `sleep` in an interactive pod. ## Batch jobs :::tip If you need to run a larger and longer computation, use one of the available [Workload Controllers][4]. We recommend running those as [Jobs][5] - this will closely watch your workload and make sure it ran to completion (exited with 0 status), shut it down to free up the resources, and restart if the node was rebooted or something else has happened. ::: 1. Use [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) to run [batch jobs](https://nrp.ai/documentation/userdocs/running/jobs/) and set resources carefully. - You can use [Guaranteed QoS](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed) for those. 2. While running a large number (> ~100) of jobs, you must choose your resource `limit` = `request`. 3. Using `sleep infinity` in Jobs is not allowed. Users running a Job with `sleep infinity` command or equivalent (script ending with “sleep”) will be banned from using the cluster. ## Long-running idle pods If you need some pods to run idle for a long time, you can use the [Deployment controller](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment). 1. Use [**`Deployment`**](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) if you need a [long-running pod](https://nrp.ai/documentation/userdocs/running/long-idle/). 2. Make sure you set minimal requests and proper limits for those to get the [Burstable QoS](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-burstable). 3. Such a deployment can not request a GPU. 4. Deployments are automatically deleted after 2 weeks (unless the namespace is added to the exceptions list and runs a permanent service). ## Workloads purging (2 weeks max runtime) 1. A periodic process removes workloads (deployments) **older than 2** weeks to free up resources and maintain system efficiency. 2. If you need to run [some permanent services](https://nrp.ai/documentation/userdocs/running/ingress/) beyond two weeks, contact admins in [Nautilus Support](https://nrp.ai/contact) and ask for an exception. Please provide an estimated period of service functioning and a brief description of what the service does. 4. [Long idle pods](https://nrp.ai/documentation/userdocs/running/long-idle/) can't be added to the exceptions list, since those are considered temporary and we need to be sure those are cleaned when not needed. 5. You will receive 3 notifications if your workload is not on the exception list. After that, your workload will be deleted. Any data in persistent volumes will remain. ## Requesting GPUs 1. When you [request GPUs for your pod](https://nrp.ai/documentation/userdocs/running/gpu-pods), nobody else can use those until you stop your pod. You should only schedule GPUs that you can actually use. 2. Check the [GPU dashboard](https://grafana.nrp-nautilus.io/d/dRG9q0Ymz/k8s-compute-resources-namespace-gpus) for your namespace to make sure the utilization is above 40%, and ideally is close to 100%. 3. The only reason to request more than a single GPU is when your GPU utilization is close to 100% and you can leverage using more GPUs. 4. GPUs are a limited resource shared by many users. If you plan on deploying large jobs (>50 GPUs) please present a plan in [Nautilus Support][3]. :::caution[Access Request for A100 GPUs] - To prevent misuse, the default quota for A100 GPUs is set to zero for all namespaces. - To request access, complete the [A100 access](https://nrp.ai/reservations) request form online with the details of your workflow. ::: ## Data purging 1. Please [purge](https://nrp.ai/documentation/userdocs/storage/purging/) any data you do not need. You should clean up your storage at regular intervals. 2. NRP resources should not be used as archival storage. You can only store the data actively used for computations. 3. Any volume that was **not accessed for 6 months can be purged without notification**. [1]: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#specify-a-memory-request-and-a-memory-limit [2]: https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit [3]: https://nrp.ai/contact [4]: https://kubernetes.io/docs/concepts/ [5]: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ [6]: https://nrp.ai/userdocs/running/jobs/ --- # Deployed Services - URL: https://nrp.ai/documentation/userdocs/start/resources/ - Section: User Guide > Start - Description: The services cluster admins already run on Nautilus, and how to get accounts for them. Although you can run your own containers, there are services and resources already deployed by cluster admins that you can use without creating those yourself. :::caution Most services require additional registration. **[Contact us](https://nrp.ai/contact) to set up your account if the service you need requires login, but has no registration form.** ::: `Stable` - Service is **generally supported**. You can report issues with the service. `Unsupported` - Service is provided **as is**. You can report the issues but those will be checked with lowest priority. The service may be removed at any time. `Experimental` - The service is added **for testing**. Likely to have bugs or change at any time. Also may be removed at any time. Live statuses for major services are available in [**Services Status**](https://nrp.ai/services-status/). **Please report broken services in [Nautilus Support](https://nrp.ai/documentation/userdocs/start/support/).** ## Computations - [`Stable` JupyterHub][1] [(Docs)](https://nrp.ai/documentation/userdocs/jupyter/jupyterhub-service) - [`Stable` Coder: remote development environment][13] [(Docs)](https://nrp.ai/documentation/userdocs/coder/coder) - [`Stable` Kubevirt: Virtual Machines][2] - [`Unsupported` WebODM (Web Open Drone Map): Drone Images stitching][4] - [`Unsupported` SuperSplat: 3D Gaussian Splatting][9] ## Data sharing and collaboration tools - [`Stable` GitLab: Code and containers repository][5] - [`Stable` Overleaf: LaTeX collaboration][12] - [`Stable` EtherPad: Notebooks][3] - [`Stable` BentoPDF: In-browser PDF editing and manipulation][15] - [`Stable` Jitsi: Video conferencing][10] - [`Stable` Nextcloud: File sharing][6] - [`Stable` Draw.io: Online flowchart and diagrams][18] - [`Stable` Yopass: One-time secret sharing][16] - [`Experimental` LanguageTool][17]: Use **`https://languagetool.nrp-nautilus.io/v2`** in the **`Server > Custom > Server URL`** menu within the settings of each addon or plugin (such as Chrome or Microsoft Word). - [`Unsupported` Hedgedoc: collaborative markdown editor][14] - [`Unsupported` SyncThing: File sync][8] ([Contact us to set up](https://nrp.ai/contact)) ## Artificial intelligence and LLM - [`Stable` NRP-Managed LLMs and services](https://nrp.ai/documentation/userdocs/ai/llm-managed) ## Monitoring - [`Stable` Grafana: Observability and monitoring platform][19] - [`Stable` Plausible Analytics: Privacy-first web analytics][20] - [`Stable` PerfSONAR][11] - [`Unsupported` Traceroute tool][7] ## Others - [`Stable` NetBox: Infrastructure tracking](https://netbox.nrp-nautilus.io) ## Kubernetes operators (CRDs) ### Databases - `Stable` [PostgreSQL (zalan.do)](https://nrp.ai/documentation/userdocs/running/postgres) - `Stable` [Elasticsearch](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elasticsearch-specification.html) - `Stable` [Prometheus](https://prometheus-operator.dev) (users are allowed to create [`ServiceMonitors`](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.ServiceMonitor)) - `Experimental` [ClickHouse](https://nrp.ai/documentation/userdocs/running/clickhouse) - `Experimental` [Nebula graph](https://www.nebula-graph.io) - `Experimental` [ScyllaDB](https://operator.docs.scylladb.com/stable/resources/scyllaclusters/index.html) ### Compute - `Stable` [Ray](https://nrp.ai/documentation/userdocs/running/ray-cluster) - `Stable` [Dask](https://nrp.ai/documentation/userdocs/running/dask-cluster) - `Stable` [Kubeflow](https://nrp.ai/documentation/userdocs/running/kubeflow) ### Others - `Stable` [Cert-manager](https://nrp.ai/documentation/userdocs/running/ingress#auto-renewing-the-certificate) - `Stable` [Sealed secrets](https://github.com/bitnami-labs/sealed-secrets?tab=readme-ov-file#usage) - `Stable` [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) - `Stable` [Admiralty federation](https://admiralty.io) - `Experimental` [Clabernetes Containerlab](https://containerlab.dev/manual/topo-def-file/) - `Experimental` [keda.sh autoscaling](https://keda.sh) [1]: https://jupyterhub-west.nrp-nautilus.io [2]: https://nrp.ai/documentation/userdocs/running/virtualization-general [3]: https://etherpad.nrp-nautilus.io [4]: https://webodm.nrp-nautilus.io [5]: https://gitlab.nrp-nautilus.io [6]: https://nextcloud.nrp-nautilus.io [7]: https://traceroute.nrp-nautilus.io [8]: https://syncthing.net [9]: https://supersplat.nrp-nautilus.io [10]: https://jitsi.nrp-nautilus.io [11]: https://perfsonar.nrp-nautilus.io/maddash-webui/ [12]: https://overleaf.nrp-nautilus.io [13]: https://coder.nrp-nautilus.io [14]: https://hedgedoc.nrp-nautilus.io [15]: https://bentopdf.nrp-nautilus.io [16]: https://credentials.nrp-nautilus.io [17]: https://languagetool.org/services [18]: https://drawio.nrp-nautilus.io [19]: https://grafana.nrp-nautilus.io [20]: https://nrp-site-analytics.nrp-nautilus.io --- # Glossary - URL: https://nrp.ai/documentation/userdocs/start/glossary/ - Section: User Guide > Start - Description: Definitions of the Kubernetes, container, and storage terms used throughout these docs. ## Nautilus Glossary This is an abridged list of terminology that Nautilus users may find useful. These glossary entries explain terminology used in the Nautilus User Documentation and Tutorials. - **Artificial Intelligence** or **AI** refers to the development of computer systems that can perform tasks that typically require human intelligence. These tasks include learning from experience (machine learning), understanding natural language, recognizing patterns, solving problems, and making decisions. The goal of artificial intelligence is to create machines that can perform tasks intelligently, simulating human cognitive functions. - **Command** or **Unix command** is a text-based instruction or a program that is entered by a user in a Unix-like operating system's [command-line interface (CLI)](#commandLineInterface) to perform various tasks. Unix commands are used to interact with the operating system, manipulate files and directories, execute programs, manage processes, and perform system administration tasks. The command-line interface in Unix is often referred to as the shell. - **Command Line Interface** or **CLI** is a text-based user interface used to interact with a computer or software by entering commands into a terminal or console, often called a shell. - **Container** See [Software Container](#softwareContainer) - **CUDA** CUDA stands for **C**ompute **U**nified **D**evice **A**rchitecture. It is a parallel computing platform and programming model developed by NVIDIA for performing general-purpose computing on NVIDIA GPUs (Graphics Processing Units). - **Deployment** In Kubernetes, a Deployment is a resource object that manages the deployment and scaling of a set of replica [pods](#pods). Deployments provide a declarative way to define the desired state for an application and ensure that the current state matches the desired state. For example, if you wish to have three [pods](pods) running their services (i.e., [software containers](#softwareContainer)) at all times, instead of launching three separate [jobs](jobs), you would launch a deployment with three replica pods. - **Directory** In Unix-like operating systems, a directory is a file system container that holds files and other directories. It serves as a way to organize and structure the file system hierarchy. Directories are sometimes referred to as folders in other operating systems. - **Force Delete** In computer science, the term "force delete" typically refers to a method of deleting files or directories (folders) that cannot be removed through the usual or standard deletion process. It's important to exercise caution when using force delete commands, as they bypass certain safety checks and can result in the loss of important data. In [Kubernetes](#kubernetes), when you want to stop or delete a running pod (and its associated containers), you typically use the `kubectl delete` command. - **Graphical User Interface** or **GUI** is a type of user interface that allows users to interact with electronic devices or software through graphical elements such as icons, buttons, and windows. In a GUI, users can manipulate and control the system or application by using visual elements, typically with the help of a pointing device such as a mouse or a touchpad. - **GUI** See [Graphical User Interface](#graphicalUserInterface) - **Init Container** In Kubernetes, an "**init container**" is a special type of container that runs before the main application containers in a pod. Init containers are used to perform setup or initialization tasks before the application containers start. They are designed to run to completion (i.e., run to success or failure) before the application containers begin their execution. - **Job** In [Kubernetes](#kubernetes), a job is a finite or batch task that runs to completion. Unlike other Kubernetes controllers like [Deployments](#deployment), which are designed to manage and maintain a desired number of replica pods continuously, a Job is intended for short-lived tasks that perform a specific job and then terminate. - **Kubernetes** is an open-source container orchestration platform designed to automate the deployment, scaling, and management of containerized applications. Originally developed by Google and later donated to the Cloud Native Computing Foundation (CNCF), Kubernetes provides a robust and extensible platform for managing containerized workloads and services. - **Machine Learning** or **ML** is a subset of [artificial intelligence (AI)](#artificialIntelligence) that focuses on the development of algorithms and statistical models that enable computers to perform tasks without explicit programming. The primary goal of machine learning is to enable computers to learn from data and improve their performance over time without human intervention. - **Namespace** In [Kubernetes](#Kubernetes), a namespace is a virtual cluster within the physical cluster. It provides a way to partition and organize resources within a Kubernetes cluster, allowing multiple users, teams, or projects to share the same cluster without interfering with each other. Namespaces help in creating isolated environments and preventing naming conflicts between different resources. - **Nautilus** is a distributed, hyper-converged GPU/CPU cluster made of heterogeneous nodes that include a variety of GPU cards, CPUs, storage and networks. - **Orchestration** In Kubernetes, orchestration refers to the automated coordination and management of containerized applications, their deployment, scaling, and operation within a cluster, such as Nautilus. - **Persistent Volume Claim** or **PVC** is a resource that represents a request for storage by a user or a pod. It is part of the storage abstraction in Kubernetes, providing a way for applications to request and use persistent storage resources without needing to know the details of the underlying storage infrastructure. PVCs can be requested by a pod, created by a user, mounted easily to a container and shared by multiple users or processes. - **Pod** In Kubernetes, a pod is the smallest and simplest unit in the deployment model. A pod represents a single instance of a running process in a cluster, and it encapsulates one or more containerized applications. At a minimum, a pod must run some kind of operating system, such as Ubuntu Linux. - A **Service** is an abstraction that defines a logical set of pods and a policy by which to access them. Services enable a dynamic and scalable way for different parts of an application to communicate with each other within a Kubernetes cluster. For example, they may provide a stable endpoint (IP address and DNS name) that allows other services or external users to reach the pods associated with the service. - **Shell** In Unix, a shell is a command-line interpreter or a command processor that provides a user interface for interacting with the Unix or Unix-like operating systems (for example, Mac OS X). The shell is a crucial component of the operating system, responsible for interpreting user commands, executing programs, and managing the interaction between the user and the operating system kernel. - **Software Container** A software container is a lightweight, standalone, and executable package that includes everything needed to run a piece of software, including the code, runtime, system tools, libraries, and settings. Containers provide a consistent and reproducible environment, ensuring that the software runs consistently across different computing environments. - **Stateless** In computer engineering and software development, the term _stateless_ refers to a system or application that does not retain information about the state or status of a user or entity between sessions or interactions. In the case of Nautilus, [pods](#pod) are stateless - each request to instance a pod is treated as an independent and self-contained transaction, and the cluster does not store any information about the previous pod or its characteristics. - **Unix** is a powerful and versatile multiuser, multitasking operating system originally developed in the 1960s and 1970s at Bell Labs (AT&T). It is known for its stability, security, and support for a wide range of applications and services. Unix has had a significant impact on the development of operating systems and has influenced many modern operating systems, including Linux and MacOS. --- # FAQ - URL: https://nrp.ai/documentation/userdocs/start/faq/ - Section: User Guide > Start - Description: Quick answers to common problems on Nautilus, from stuck Terminating pods to storage and access questions. **My pod is stuck Terminating.** > This happens for a few reasons, such as: > * The node running your pod went offline. The pod will finish terminating once the node is back online. > * The storage attached to the pod can't be unmounted. > * Due to the high load on the node, your pod termination process could not be completed. > In all these cases, you should ask a cluster admin in [Nautilus Support](https://nrp.ai/contact) chat to look at your pod, or just wait for somebody to fix it. > :::danger[Do not force-delete your pod] > Do not use `kubectl delete --grace-period=0 --force` to delete stuck pods! > It will keep resources attached to the node for an indefinite period and will require the rebooting of the node (if you happen to know which node was it). > ::: -------- **I tried to use `nvprof` in my GPU pod and got an error.** > There is a [vulnerability](https://nvidia.custhelp.com/app/answers/detail/a_id/4738) in NVIDIA drivers still not fixed, and this feature is disabled by default. Enabling it requires too much effort, so for now we keep it default. Hopefully it will be fixed soon. -------- **How do I acknowledge support from NRP / Nautilus in research papers?** 1. Please acknowledge the NRP NSF grants in the format specified by the [AUP](https://nrp.ai/NRP-AUP.pdf). 2. Please cite the following paper when acknowledging the National Research Platform (NRP): [The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster](https://dl.acm.org/doi/10.1145/3708035.3736060) **BibTeX:** ```bibtex @inproceedings{10.1145/3708035.3736060, author = {Weitzel, Derek and Graves, Ashton and Albin, Sam and Zhu, Huijun and Wuerthwein, Frank and Tatineni, Mahidhar and Mishin, Dmitry and Khoda, Elham and Sada, Mohammad and Smarr, Larry and DeFanti, Thomas and Graham, John}, title = {The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster}, year = {2025}, isbn = {9798400713989}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3708035.3736060}, doi = {10.1145/3708035.3736060}, abstract = {The National Research Platform (NRP) represents a distributed, multi-tenant Kubernetes-based cyberinfrastructure designed to facilitate collaborative scientific computing. Spanning over 75 locations in the U.S. and internationally, the NRP uniquely integrates varied computational resources, ranging from single nodes to extensive GPU and CPU clusters, to support diverse research workloads including advanced AI and machine learning tasks. It emphasizes flexibility through user-friendly interfaces such as JupyterHub and low level control of resources through direct Kubernetes interaction. Critical operational insights are discussed, including security enhancements using Kubernetes-integrated threat detection, extensive monitoring, and comprehensive accounting systems. This paper highlights the NRP’s growing importance and scalability in addressing the increasing demands for distributed scientific computational resources.}, booktitle = {Practice and Experience in Advanced Research Computing 2025: The Power of Collaboration}, articleno = {69}, numpages = {5}, keywords = {Distributed Computing, Kubernetes, High Throughput Computing, Artificial Intelligence}, location = {}, series = {PEARC '25} } ``` --- # Asking Effective Questions in Support Chat - URL: https://nrp.ai/documentation/userdocs/start/support/ - Section: User Guide > Start - Description: How to prepare and ask a support question in Nautilus chat so admins can answer it fast. To ensure that our support team can provide you with the most accurate and efficient assistance, we've outlined some guidelines for asking effective questions in our support chat. To reach out to us please use [Nautilus Support](https://nrp.ai/contact). ## Before You Ask Before reaching out to us, please take a moment to gather the necessary information about the issue you're experiencing. This will help us quickly identify the problem and provide a more accurate solution. ## Essential Information to Provide When asking a question in support chat, please provide the following essential information: - **Namespace and Pod Name**: If the issue is related to a specific pod in your cluster, please specify the namespace and pod name. This information helps us pinpoint the exact resource that's experiencing the problem. - **Minimal Reproducible Example**: If the issue is not specific to a particular pod or namespace, please provide a minimal reproducible example (MRE) that demonstrates the problem. MREs should be a concise, self-contained code snippet that reproduces the issue. - **Service**: If you are using a service like JupyterHub or Coder, mention this in chat. Share the url you are using to access this service. Provide the namespace in which this service is running/meant to run. Note that [https://jupyterhub-west.nrp-nautilus.io](https://jupyterhub-west.nrp-nautilus.io) runs in the namespace **jupyterlab**. :::note [https://jupyterhub-west.nrp-nautilus.io](https://jupyterhub-west.nrp-nautilus.io) runs in the namespace **jupyterlab**. Share either with the admin when assking for help. If you are using a different JupyterHub, please contact the namespace admin where the hub is runnning for assistance. ::: ## Gathering Information with kubectl To help you gather more information about the issue, we recommend using the `kubectl` command-line tool. Here are some tips on how to use `kubectl` to check pod logs and status: ### Checking Pod Status To check the status of a pod, use the following command: ```bash kubectl describe pod -n ``` This command will provide you with a detailed overview of the pod's status, including: - Pod events - Container status - Volumes and mount points Look for any error messages or unusual conditions that may indicate the cause of the issue. ### Checking Pod Logs To check the logs of a pod, use the following command: ```bash kubectl logs -n -c ``` Replace `` with the name of the container you want to inspect. If the pod has only one container, you can omit the `-c` flag. You can also use the following flags to customize the log output: - `-f` to follow the log output in real-time - `--tail` to specify the number of lines to display from the end of the log - `--since` to specify the timestamp from which to display logs For example: ```bash kubectl logs -n -c -f --tail 100 ``` This command will display the last 100 lines of the log output from the specified container and follow the log output in real-time. ## What to Expect Without Essential Information If you don't provide the namespace and pod name or a minimal reproducible example, our support team can only provide a best guess based on incomplete information. While we'll do our best to help, we may not be able to provide an accurate diagnosis or solution. This can lead to unnecessary back-and-forth, resulting in delays and inefficiencies. ## Examples of Good Questions Here are some examples of well-crafted questions that include the necessary information: - "I'm seeing an error in my pod `my-pod` in the `default` namespace. The logs indicate a connection issue. Can you help me troubleshoot?" - "I'm experiencing a problem with my deployment. Here's a minimal reproducible example that demonstrates the issue: [insert code snippet]. Can you help me identify the root cause?" - "I've checked the pod logs and status using `kubectl describe pod` and `kubectl logs`, but I'm still unsure what's causing the issue. Here's the output from both commands: [insert output]. Can you help me interpret the results?" ## Tips for Asking Questions To ensure that your question is answered efficiently, follow these additional tips: - Be concise and clear in your question. - Provide any relevant error messages or logs. - Use proper formatting to make your question easy to read. For example, use tripple backticks \`\`\` around your code snippets like so: ```` ``` import psycopg2 conn = psycopg2.connect("dbname=mydatabase user=myapp password=mypassword") cur = conn.cursor() cur.execute("SELECT version();") print(cur.fetchone()) cur.close() conn.close() ``` ```` :::note By following these guidelines, you'll be able to ask effective questions that help our support team provide you with accurate and efficient solutions. Thank you for helping us help you! ::: --- # Introduction - URL: https://nrp.ai/documentation/userdocs/tutorial/introduction/ - Section: User Guide > Tutorials - Description: Where to start with the tutorials, what order to take them in, and when to jump straight to the Running pages. In these tutorials you will learn the basic skills needed to access resources associated with the Nautilus Cluster. These resources can include GPU and CPU cycles, storage and even software. For novice users, we recommend that you step through all the tutorials, especially tutorials on launching pods, requesting a storage volume, and using Docker. ## What is Nautilus? Nautilus is a distributed, hyper-converged cluster where storage, compute and networking are tightly integrated into a single unified system. What distinguishes Nautilus from other such systems is that Nautilus is distributed across dozens of locations. Individual Nautilus nodes are comprised of high-performance GPU servers combined with high-speed data transfer-and-storage nodes, and additional storage nodes that serve as data reservoirs...all stitched together with very fast networks. Unlike standalone GPU resources, jobs sent to Nautilus can run anywhere in this vast cluster, giving the user flexibility to run single tasks or up to many tasks across multiple nodes and locations while Kubernetes manages the workload. ## Why use Nautilus? Nautilus has a huge variety of both standard GPUs (for example: NVIDIA RTX2080-Ti, RTX3090, RTX4090 and A10) and high-precision, high-memory GPUs (for example: NVIDIA A100 Tensor Core). It also has the ability to download large datasets very quickly and store them near any computation you might launch. Since Nautilus is a hyper-converged cluster which supports a high-degree of virtualization, the user does not need to worry about where their data reside or where their jobs will run. Furthermore, because software runs in containers, it is scaleable, repeatable and customizable without the hassle of system configuration dependencies. Software containers provide portability, efficiency, scaleability, consistency, speed, and security. Nautilus makes managing the data, software and hardware required for training of machine learning easier. ### But how much does it cost? Perhaps most importantly, for individuals associated with institutions that participate in the InCommon identity and access management service, and who are conducting sponsored research in artificial intelligence or machine learning, Nautilus is FREE to use (subject to [policy restrictions](https://nrp.ai/documentation/userdocs/start/policies/) and fair use). ## Steps to Successful Nautilus Use There are several key steps which you must go through to get access to Nautilus. They are: 1. Establish an account 2. Configure your local system to interact with Nautilus 3. Be promoted to User or Admin 4. Read and understand the policies 5. For beginners, complete the [Tutorials](./basic) :::caution Familiarity with the Unix command line is the single biggest predictor of success with Nautilus. For most users who are unfamiliar with the Unix shell, this will prevent them from being able to use Nautilus, as it has no graphical user interface or desktop application. Users who do not understand the basic principles of Unix, lack the ability to navigate a Unix file system through a terminal window, do not understand Unix commands, have no experience with scripting or scripting languages, or have never written code before are highly discouraged from attempting to use Nautilus without first having a basic grasp of these skills. ::: ## What if I don't think I have time to learn a bunch of new skills? In addition to being able to launch jobs using Kubernetes on the cluster, Nautilus provides a simpler interface called a [Jupyterlab Notebook](https://jupyter.org). This is for people who just want a easy way to interact with the cluster without any local installations or configurations. JupyterLab is a web-based interactive development environment for notebooks, code, and data. Its flexible interface allows users to configure and arrange workflows in data science, scientific computing, and machine learning. There are many places where you can try [JupyterLab](https://jupyter.org/try) online, but since our Jupyterlab is backed up by the Nautilus cluster, your Notebook and its code can run on as many as eight GPUs working as a single computational engine. Python is the main language used on JupyterLab. While Python is an essential skill for machine learning, it's also easy to learn. If you want to try Nautilus, but without learning Unix, you can start by [learning Python](https://www.python.org/about/gettingstarted/). And by the way, no need to install Python locally, just reach out to the community on [Nautilus Support](https://nrp.ai/contact) to request access to the Jupyterlab running on Nautilus. It will run in your browser without any local software installations. ## Getting Help Nautilus is a community-based cluster, managed by a small group of NSF-funded administrators. Every effort has been made to provide a dependable computational platform, but Nautilus is also largely self-service and community supported. Users are expected to learn-by-doing, and generally rely on community-based resources for troubleshooting. A diverse and active user community can be found by joining the Nautilus Support channel. Having a Nautilus account is not required for access to Nautilus Support, but having access to Nautilus Support is required for joining Nautilus. Find out more about this by [joining the chat channel](https://nrp.ai/contact) today. ## Now, let's get started... [Get Access](https://nrp.ai/documentation/userdocs/start/getting-started/) --- # Docker and Kubernetes - URL: https://nrp.ai/documentation/userdocs/tutorial/docker/ - Section: User Guide > Tutorials - Description: Build a container image on your own machine and get it onto Nautilus. ## Introduction Learning to use Docker for Nautilus enables you to take advantage of containerization technology to streamline the deployment, management, and scalability of your applications on the cluster. It also aligns with modern DevOps practices and helps you leverage the full capabilities of the Nautilus infrastructure. ## Prerequisites You will need the ability to install applications on your computer running a supported operating system (OS), such as Windows, Mac OS X or Linux (Chromebooks are not supported). You should also be familiar with the basic concepts of software containers (images) and [Docker Hub](https://hub.docker.com). ### Installing docker Docker container image is an easy way to package some files together and run as an isolated environment on virtually any computer in the world - from your own laptop to a supercomputer. The most popular, but not the only one, tool to run containers is [Docker](https://www.docker.com). There is a [nice guide](https://www.docker.com/get-started) on installing docker on your local machine and [getting started with it](https://www.docker.com/101-tutorial). ### Building image There are millions of already existing containers available for free on [docker hub](https://hub.docker.com) (use the search field to search for the software you need). In case you want something that doesn't exist there, or want to extend the already existing container, it's easy to do so. To build a new container, you first need to create a Dockerfile. Let's say we want a container which has python3 in it, and also has the geojson library installed to parse some geojson files and the vim text editor. First we search by keyword: . Usually the most popular images appear on top. The [python repository](https://hub.docker.com/_/python) looks good to us, and we go to see which tags are available from the repository. _Container tag_ is a flavor of a container - it can tell us which version of the software it contains, which variant of software setup is used if there are many, etc. The default tag is `latest` if you don't specify any, but it's advised to specify at least the version so that your workflow is not suddenly broken by `:latest` image version update. The tag is specified after the container name, f.e. `python:3.9`. Let's say we browsed the [python tags](https://hub.docker.com/_/python?tab=tags) page and found that we want to base our own image on the [3.9 version](https://hub.docker.com/layers/python/library/python/3.9/images/sha256-bdc5af48b5f7bf9fe141057d17dbdfbfadc1c348dbb7292f8856d993bd66af63?context=explore). On the specific tag page we can see the Dockerfile commands which were used to build the image (not for all images). We could copy the commands and modify those to our own needs, but instead we might simply derive our container from the one already existing. To do that we start our Dockerfile with `FROM` directive: ``` FROM python:3.9 RUN apt-get update && \ apt-get install -y vim && \ pip install geojson ``` The reason to put 3 comands in a single RUN is that we want those to be in a single container _layer_, which will speed up downloads and minimise the image size. Here's the [best practices guide](https://www.docker.com/blog/intro-guide-to-dockerfile-best-practices/). Save this as `Dockerfile` and run the command (in case you have docker installed locally): ``` docker build . -t my-first-container:latest ``` You will see docker pulling the existing container _images_ from docker hub, and then executing the RUN directive you added to Dockerfile. Each such directive creates a separate _layer_ in the image, and layers are cached - this way you can have multiple python containers, and they all will share the base layers. Now type `docker images` and see that your image `my-first-container` was created and is saved in your local docker _repository_. ### Running image Now we can actually run the image. A single image can be ran multiple times, and all changes to the files will only be visible while the container is running once it stops, all new data **will be lost** unless it's placed in a [persistent volume](https://docs.docker.com/storage/volumes/). Run: ``` docker run -it --rm -v $PWD:/code my-first-container bash ``` This will start an instance of our container locally on your machine, and also the `-v $PWD:/code` mounted the current directory into /code inside the container. You can then execute `cd /code` inside the container and run some python code. Also note that geojson is installed: ``` root@19d104e34ca6:/# pip list Package Version ---------- ------- geojson 2.5.0 pip 21.0.1 setuptools 53.0.0 wheel 0.36.2 ``` To exit, execute `exit`. `--rm` told docker to kill and remove the container once we exist from it. You can run in detached mode by running instead: ``` docker run -d -v $PWD:/code my-first-container sleep infinity ``` This will start the container, but not exec into it. To exec, run `docker ps` to see running containers (not either the Container ID or the Name of your new container), and then exec into it: ``` docker exec -it bash ``` This will execute bash inside the container. YOu can enter and exit from it multiple times. To destroy, run: ``` docker rm -f ``` ### Exporting image For now the container image we've built is stored in our local docker registry. To make it available to the kubernetes cluster we need to put it somewhere globally available. You can register on docker hub and [upload there](https://docs.docker.com/docker-hub/), or use [our GitLab instance](https://nrp.ai/documentation/userdocs/development/gitlab/) and create a repository. Our GitLab is free, has unlimited space and is much faster than Docker Hub. Also for larger images it's possible to build those directly in our cluster and not on your local computer, which will significantly increase the speed and upload time. But you're free to upload the images from your local machine if you wish. --- # Basic Kubernetes (k8s) Tutorial - URL: https://nrp.ai/documentation/userdocs/tutorial/basic/ - Section: User Guide > Tutorials - Description: Hands-on basics: first kubectl commands, a pod, a deployment, and querying cluster status. In this tutorial, you will be introduced to basic k8s commands, how to launch simple pods and deployments, as well as interact with the cluster to query its status and the status of your processes running in the cluster. You will also see your first example of a YAML file. :::note[Question?] Throughout these tutorials, you may see a "Question?". This is an opportunity for you to pause and reflect, or answer a quick self-assessment that can lead to deeper understanding of the materials. ::: ## Prerequisites This section assumes you've completed the [quickstart](https://nrp.ai/documentation/userdocs/start/getting-started/) section. ## Learning Objectives 1. You will understand the basic format of k8s commands. 2. You will be able to use k8s commands to interact with the cluster. 3. You will have a basic understanding of YAML and you will be able to use that to create a simple pod. 4. You will have an understanding of the difference between an pod and a deployment. 5. You will have a basic understanding of the "stateless" nature of a pod and how deployments can be used to specify an ideal state for your pods or software containers (running inside your pods). :::tip[Setting your namespace] Since you only have permissions to run in namespaces of which you are a member, you must specify your namespace. For the purposes of this tutorial, it is recommended that you set your namespace globally with this command: ```bash kubectl config set-context nautilus --namespace= ``` Some users may operate concurrently across several namespaces, and k8s allows you to specify which in each `kubectl` command by adding `-n `, regardless of whether you have set your context as above. Most users save the keystrokes by setting the context for themselves and deviate only when necessary, rather than specifying each time. ::: ## Explore the system Now that you understand how to set your namespace, let's begin to explore the system. ### List cluster nodes The Nautilus Cluster is widely geographically distributed and highly heterogenous. You can get a sense of the types of nodes in the system by typing: ```bash kubectl get nodes ``` **Please note**: You likely won't have access to all the nodes listed, as some are reserved. ### List processes running in your namespace There are three categories of processes we will examine: - [pods](https://nrp.ai/documentation/userdocs/start/glossary) - [deployments](https://nrp.ai/documentation/userdocs/start/glossary) - [services](https://nrp.ai/documentation/userdocs/start/glossary) Listing the categories running in k8s follows a similar format. ```bash kubectl get ``` Right now you probably don't have anything running in the namespace, and these commands will return `No resources found in ... .` but you will revisit these commands as we step through the tutorials as a way of checking on status. #### List pods List all the pods in your namespace ```bash kubectl get pods ``` #### List deployments List all the deployments in your namespace ```bash kubectl get deployments ``` #### List services List all the services in your namespace ```bash kubectl get services ``` ## Setup your first pod with YAML Let’s create a simple generic pod, and then login into it. ### A simple YAML file Create the `pod1.yaml` file with the following contents by copy-pasting: ```yaml // pod1.yaml apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: mypod image: ubuntu resources: limits: memory: 200Mi cpu: 200m requests: memory: 200Mi cpu: 200m command: ["sh", "-c", "echo 'Im a new pod' && sleep infinity"] ``` **Reminder**: Indentation is important in YAML, just like in Python :::tip[A simple way to create a file un Unix] There are many ways to create a file and put data into it in Unix. One of the easiest ways is to redirect the standard input with the following command: ```bash title="Create a new file" cat > ``` You can then paste any copied text or type directly into the command line interface. Once you are done, you can `control-c` out of the input stream. This will do the job without leaving the command line interface. Try it by copying the `pod1.yaml` sample above, and then pasting it into a new file using the Unix `cat` command. ::: ### Creating YAML files dynamically Alternatively, if you don't want to create a file and are using Unix-like system, you can create YAML files dynamically like this: ```bash kubectl create -f - << EOF EOF ``` ### Launch a simple pod Making sure you are in the same file directory as your `pod1.yaml` file, type the following command: ```bash kubectl create -f pod1.yaml ``` After a few moments (as the pod is creating itself), see if you can find it: ```bash kubectl get pods ``` **Please Note**: You may see the other pods too. If it is not yet in Running state, you can check what is going on with a list of the events in your namespace: ```bash kubectl get events --sort-by=.metadata.creationTimestamp ``` Events and other useful information about the pod can be seen in `describe`: ```bash kubectl describe pod test-pod ``` :question: Where did the name _test-pod_ come from? Examine the `pod1.yaml` file to find the answer. If the pod is in Running state, we can check it's logs ```bash kubectl logs test-pod ``` Let’s log into it ```bash kubectl exec -it test-pod -- /bin/bash ``` :::tip[There's a relationship between the operating system and the command line interpreter] The last part of this command, specifying `bash` can change, depending on the operating system we choose. Keep this in mind when you deploy operating systems other than Ubuntu. ::: :::note[Question?] Did you manage to log into your pod? If yes, you are now inside the (container in the) pod! ::: :::note[Question?] Does it feel any different than a regular, dedicated node? ::: Try to create some directories and some files with content (using the `cat` command, if you like). "Hello world" will do, but feel free to be creative. ### Let's examine the pod's networking Networking inside a Kubernetes pod is crucial for facilitating communication between containers within the same pod and enabling connectivity with other pods, as well as external services. Understanding how networking works inside a pod is essential for building and deploying applications effectively in a Kubernetes environment. So, let's examine how the network is configured inside our simple pod by logging into it. :::note[Question?] Do we have the necessary tools to examine the network installed in our pod? ::: Remember that pods running inside k8s are "stateless" and since we didn't specify any software packages to be included in our [simple pod](#a-simple-yaml-file) example when we launched it, we have some work to do before we can look at the network. The package `ifconfig` is not included in our initial pod; so let’s install it. First, let's make sure our installation tools are updated. ```bash apt update ``` Now, we can use apt to install the necessary network tools. ```bash apt install net-tools ``` Now check the networking: ```bash ifconfig -a ``` :::note[Question?] What did you discover? Does the output look like you'd expect? ::: Finally, let's exit out of the pod and move on by entering the `exit` command in the command line interface of the pod, or using the keyboard shortcut 'Control-D'. ### Testing statelessness To demonstrate that pods really are stateless, we are going to shutdown our simple pod, and repeat the creation of our simple pod, then take a look at it again. ### Shutdown the originall Let's manually shut down the pod using `kubectl` ```bash kubectl delete -f pod1.yaml ``` This may take a moment, as the system will remove the pod gracefully. After a few moments, check that it is actually gone: ```bash kubectl get pods ``` :::note[Question?] Is it gone? If yes, let’s create it again: ```bash kubectl create -f pod1.yaml ``` ::: :::tip[Accessing prior commands from the CLI] Most command line interfaces store the most recent commands you have entered. You can access them easily by using your keyboard's "up arrow" which is a quick and easy way to repeat commands. ::: ### Looking at Pod1.yaml (again) Give the system a moment to create the new pod. :::note[Question?] Does it have the same IP? We can check by using the following command: ``` kubectl get pod -o wide test-pod ``` Log back into the pod: ``` kubectl exec -it test-pod -- /bin/bash ``` ::: Now, let's look for the files you created with the `cat` command. Are they where you left them? What is the status of the files your created? :::note[Question?] How does this exercise demonstrate "statelessness" and what are the implications for how I prepare for the inevitable and normal restarting of a pod? ::: ### Cleaning up Since Nautilus is a shared platform and a community resource, it's very important that we don't leave any stray processes running. So, everytime we're finished using a pod, we should make sure to clean up after ourselves. So, let’s delete explicitly the pod, using the following command: ```bash kubectl delete pod test-pod ``` ## Let’s transform our pod into a deployment You saw that when a pod was terminated, it was gone. While above we did it by ourselves, the result would have been the same if a node died or was restarted. This is normal and expected in a k8s cluster, and is actually a great feature of Kubernetes: it is highly resilient. Kubernetes constantly monitors the health of pods and containers. In the event of a failure or unresponsiveness, Kubernetes automatically restarts containers or entire pods to maintain the desired state specified by the user. In order to specify to the cluster your "desired state", the use of Deployments is recommended. You can copy-and-paste the lines below into a new file on your local system (using the `cat` command, if you like). ```yaml title="Deployment 1" // dep1.yaml apiVersion: apps/v1 kind: Deployment metadata: name: test-dep labels: k8s-app: test-dep spec: replicas: 1 selector: matchLabels: k8s-app: test-dep template: metadata: labels: k8s-app: test-dep spec: containers: - name: mypod image: ubuntu resources: limits: memory: 500Mi cpu: 500m requests: memory: 100Mi cpu: 50m command: ["sh", "-c", "sleep infinity"] ``` Now let’s start the deployment: ```bash kubectl create -f dep1.yaml ``` See if you can find it: ```bash kubectl get deployments ``` :::tip[The Deployment is just a conceptual service] It describes to the cluster the _ideal state_ of your pods. It doesn't actually do more than that. In this case, the _ideal state_ is a single replica of the container called "mypod". ::: See if you can find the associated pod: ```bash kubectl get pods ``` Once you have found the name assigned to it by the cluster, let’s log into it. ```bash kubectl get pod -o wide test-dep- kubectl exec -it test-dep- -- /bin/bash ``` You are now inside the (container in the) pod! ### Testing statelessness with deployments Create directories and files as before. Try various commands as before. Let’s now delete the pod! ```bash kubectl delete pod test-dep- ``` Is it really gone? ```bash kubectl get pods ``` What happened to the deployment? ```bash kubectl get deployments ``` Get into the new pod ```bash kubectl get pod -o wide test-dep- kubectl exec -it test-dep- -- /bin/bash ``` :::note[Question?] Was anything preserved? How might you make sure the settings, software packages, and data you need in your pod is preserved? ::: :::tip[Using the right image helps with reliability and resiliency] Ensuring that the software container you are deploying has all the necessary elements to function involves creating a well-packaged and self-contained container image. Unlike the simple examples above, you can make a robust and reliable deployment by specifying an appropriate base image, and including any system libraries, and any language-specific runtime or dependencies in your software container. ::: :::note[Question?] Examining the [`dep1.yaml`](#deployment-1) example above, what would you change to specify a different image? Keep that in mind as we progress through the tutorials. If you are curious and want to read ahead, we cover that topic in the section called "[Images](https://nrp.ai/documentation/userdocs/tutorial/images)". ::: Let’s now delete the deployment: ```bash kubectl delete -f dep1.yaml ``` Verify everything is gone: ```bash kubectl get deployments kubectl get pods ``` ## Next steps In the next [tutorial](../basic2), we will explore how to run a simple web server in a pod, and how to expose it to the outside world and scale it. --- # Scheduling and Exposing - URL: https://nrp.ai/documentation/userdocs/tutorial/basic2/ - Section: User Guide > Tutorials - Description: Take a service from one pod to several with scheduling and horizontal scaling, then expose it. In a Kubernetes cluster, orchestration refers to the automated coordination, deployment, scaling, and management of containerized applications and their associated workloads. Horizontal scaling is crucial for services hosted on a Kubernetes cluster for several reasons, and it aligns with the fundamental principles of container orchestration and cloud-native architectures. Horizontal scaling in Nautilus allows you increase the overall workload of your jobs, improve reliability and availability, use resources more efficiently, load balance across replicas, and it increases fault tolerance (no single instance failing disrupts your work). Horizontal scaling is common practice for cloud-native applications. ## Prerequisites This section builds on skills from both the [Quickstart](https://nrp.ai/documentation/userdocs/start/getting-started/) and the tutorial on [Basic Kubernetes](https://nrp.ai/documentation/userdocs/tutorial/basic). ## Learning Objectives 1. You will learn how to deploy a basic Apache service across multiple replicas. 2. You will learn how to load balance between replicas. 3. You understand how to expose services running inside of your pods to the public internet. ## Let's launch multiple web servers In this exercise, we will launch multiple Web servers. To make distinguishing the two servers easier, we will force the nodename into their homepages. Using stock images, we achieve this by using an [init container](https://nrp.ai/documentation/userdocs/start/glossary). You can copy-and-paste the lines below into a new file called `http2.yaml` (using the `cat` command to redirect the standard input). ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: test-http labels: k8s-app: test-http spec: replicas: 2 selector: matchLabels: k8s-app: test-http template: metadata: labels: k8s-app: test-http spec: initContainers: - name: myinit image: busybox command: [ 'sh', '-c', "echo '

I am ' `hostname` '

' > /usr/local/apache2/htdocs/index.html", ] volumeMounts: - name: dataroot mountPath: /usr/local/apache2/htdocs containers: - name: mypod image: httpd:alpine resources: limits: memory: 200Mi cpu: 1 requests: memory: 50Mi cpu: 50m volumeMounts: - name: dataroot mountPath: /usr/local/apache2/htdocs volumes: - name: dataroot emptyDir: {} ``` Examine the above text and try to identify what makes this different than the other YAML files we've encountered so far. Some new fields can be found under the `initContainers` tag. :::note[Question?] **What's different about the image you are running in this pod?** Previously, we ran versions of Ubuntu, and in one instance, we had to install additional software manually. How could we avoid that in the future? You might look up "[busybox](https://hub.docker.com/_/busybox)" to understand better what that image is and what it does. ::: What happens if you want to scale beyond just two replicas? Feel free to change the number of replicas (within reason) and the text it is shown in home page of each server, if so desired. :::caution Note that the "httpd" container defines the `command` to run, which is the web server in this case. If you're running some other container that does not define the command, you'd have to specify it in the `command` field (instead of `sleep infinity` in previous examples). This ensures that the container does what you expect every time it starts. ::: ### Launch the deployment: Now that you have created this new YAML file, and you understand what makes this file different, let's try it by executing the following command. ``` kubectl create -f http2.yaml ``` :::caution Since we know that containers running in Nautilus aren't exposed to the broader internet, but it's critical to remember that they _are_ exposed to other pods running in our namespace. This is feature is beneficial when thinking about deploying complex software or services in Kubernetes. ::: In order for us to examine what's happening in our deployment, we'll need to access it via another pod running within our namespace. #### Let's build a pod to examine our deployment It's an important skill to be able to iterate by modifying your existing YAML files. This is how most people develop new versions: they modify existing versions. In this mini-exercise, we are going to copy `pod1.yaml`, make some modifications using our favorite CLI text editor (e.g. vi, nano or vim), and use our modified version to peer into our pods from `http2.yaml`. Let's recall what the `pod1.yaml` file looked like: ```yaml apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: mypod image: ubuntu resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: ['sh', '-c', "echo 'Im a new pod' && sleep infinity"] ``` :::note[Question?] What resources are we requesting in this example? Are the resources requested for this pod capable of much? ::: In Kubernetes, the base unit for describing memory resources is the `byte`. However, to make it more convenient and human-readable, memory values are commonly expressed in multiples of bytes using the International System of Units (SI) prefixes. - Kilobyte (Ki): 1 Ki is equivalent to 1024 bytes. - Megabyte (Mi): 1 Mi is equivalent to 1024 KiB or 1,048,576 bytes. - Gigabyte (Gi): 1 Gi is equivalent to 1024 MiB or 1,073,741,824 bytes. - ...and so on In Kubernetes, the base unit for describing CPU resources is the "millicore" which represents one thousandth of a CPU core. The term "millicore" is often abbreviated as "mCPU" or simply "m" (as above). For example, a CPU value of "100m" means 100 millicores, which is equivalent to 0.1 CPU core. Similarly, "500m" represents 500 millicores or 0.5 CPU core. ### A Container that has curl In order for us to use another pod to peer into our load-balanced web servers, we need a container that has `curl` as a preinstalled package. It's important for us to realize that a customized container that automates what we need is much preferred for repeatability and scale (and they are often available prebuilt in places like [Docker Hub](https://hub.docker.com)). :::note[Question?] Where might we look for a basic Ubuntu container that has `curl` preinstalled? ::: In this case, we're providing you with an image that has 'curl' preinstalled. Create a YAML file with the name `pod-curl.yaml` by coping the code below. ```yaml apiVersion: v1 kind: Pod metadata: name: test-curl-pod spec: containers: - name: mycurlpod image: curlimages/curl:latest resources: limits: memory: 200Mi cpu: 200m requests: memory: 200Mi cpu: 200m command: ['sh', '-c', "echo 'Im a new curl pod' && sleep infinity"] ``` Notice that we've increased our resources and requests for memory and CPU. When using a shared Kubernetes cluster (or any shared cluster for that matter), it is wise to balance your need for resources with their availability. Asking for too many resources will decrease the chances the cluster can provide them in a reasonable timeframe. Asking for too few resources will negatively impact your ability to compute. Once this new pod is launched, you can continue with the tutorial. #### Check your web servers are running and get their IP addresses Check the pods you have, alongside the IPs they were assigned to: ``` kubectl get pods -o wide ``` Log into pod1 ``` kubectl exec -it test-curl-pod -- /bin/sh ``` Now, from inside your `curl`-capable pod, try to pull the home pages from the two Web servers; use the IPs you obtained above: ``` curl http:// < IPofPod > ``` Each pod should say something like, `"

I am test-http-76f7d84c67-j5ff4

"`where the hash of numbers and characters is unique to each pod you are querying. ## Load balancing Having to manually switch between the two Pods is obviously tedious. What we really want is to have a single logical address that will automatically load-balance between them. You can copy-and-paste the lines below. ```yaml title="svc2.yaml" apiVersion: v1 kind: Service metadata: labels: k8s-app: test-svc name: test-svc spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: k8s-app: test-http type: ClusterIP ``` Let’s now start the service: ``` kubectl create -f svc2.yaml ``` Look up your service, and write down the IP it is reporting under: ``` kubectl get services ``` Log into pod1 ``` kubectl exec -it test-curl-pod -- /bin/sh ``` Now try to pull the home page from the service IP: `curl http://*IPofService*` Try it a few times…by repeating the command (hint: the ↑ on your keyboard will toggle through previous commands). Which Web server is serving you? Does it matter? How could you imagine using a `service` like this to balance the load across multiple pods or deployments? Note that you can also use the local DNS name for this (from pod1) `curl http://test-svc..svc.cluster.local` :::note[Namespaces?] Do you remember what namespace you are using? Be sure to note your namespace in order to access the service. ::: ## Exposing public services Sometimes you have the opposite problem; you want to export resources of a single node to the public internet. The above Web services only serve traffic on the private IP network LAN. If you try curl from your laptops, you will never reach those Pods! What we need, is set up an Ingress instance for our service. ```yaml title="ingress.yaml" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: test-ingress spec: ingressClassName: haproxy rules: - host: test-service.nrp-nautilus.io http: paths: - backend: service: name: test-svc port: number: 80 path: / pathType: ImplementationSpecific tls: - hosts: - test-service.nrp-nautilus.io ``` Launch the new ingress ``` kubectl create -f ingress.yaml ``` You should now be able to fetch the Web pages from your browser by opening . :::note[HTTPS and SSL] Note that SSL termination is already provided for you. More information is available in [Ingress section](https://nrp.ai/documentation/userdocs/running/ingress). ::: You can now delete the deployment: ``` kubectl delete -f http2.yaml kubectl delete -f svc2.yaml kubectl delete -f ingress.yaml kubectl delete -f pod-curl.yaml ``` --- # Scheduling Workloads in Nautilus - URL: https://nrp.ai/documentation/userdocs/tutorial/scheduling/ - Section: User Guide > Tutorials - Description: Learn how to schedule and optimize the placement of pods in the Nautilus Cluster, including using node selectors, resource requests, and geographical preferences. In Kubernetes, scheduling refers to the process of assigning pods to nodes in a cluster based on various factors such as resource requirements, node capacity, and other constraints. The Kubernetes scheduler is responsible for determining where and how to run pods within the cluster. ❗ While you can run jobs without any special node selectors, understanding this section will allow you to better optimize the placement of your workloads and significantly increase computational performance. You can request more performant CPUs, GPUs with more memory, faster network links, and even select nodes in a specific geographical region to optimize latency to your storage. ## Prerequisites This section builds on skills from both the [Quickstart](https://nrp.ai/documentation/userdocs/start/getting-started/) and the tutorial on [Basic Kubernetes](https://nrp.ai/documentation/userdocs/tutorial/basic). ## Learning Objectives 1. You will learn how to query the cluster to view high-level node availability in real time. 2. You will understand how node capabilities (e.g. GPU type, network speed, geography) are exposed through labels. 3. You will learn how to target specific node features without requiring direct access to individual nodes. 4. You will be able to enforce or prefer node types and resource requirements within a pod `yaml` file. ## Explore the system Let's start by looking at what's available in the system. You have already seen the list of all nodes: ``` kubectl get nodes ``` This is a very long list — and growing. While you can see basic node information, Nautilus intentionally limits direct access to detailed node configuration. > ⚠️ Note on permissions > > Nautilus users have list-only access to nodes. > > - ✅ kubectl get nodes > - ✅ kubectl get nodes -L > - ✅ kubectl get node -l > - ❌ kubectl get node > - ❌ kubectl describe node > - ❌ kubectl get nodes -o yaml > > All user-relevant scheduling information is exposed through node labels, which are safe to query and can be used directly in pod scheduling. ### Viewing node capabilities with labels You may view the full labels of a node through the following (change `` to the hostname of the node you are concerned with, e.g., `node-1-1.sdsc.optiputer.net`): ``` kubectl get nodes --field-selector 'metadata.name=' -o go-template='{{range $k, $v := (index .items 0).metadata.labels}}{{$k}}={{$v}}{{println}}{{end}}' ``` Examples of commonly usable labels (not exhaustive and variable):
Open ``` cpu-feature.node.kubevirt.io/adx=true cpu-feature.node.kubevirt.io/aes=true cpu-feature.node.kubevirt.io/avx=true cpu-feature.node.kubevirt.io/avx2=true cpu-feature.node.kubevirt.io/fma=true cpu-feature.node.kubevirt.io/sse=true cpu-feature.node.kubevirt.io/sse2=true cpu-feature.node.kubevirt.io/sse4.1=true cpu-feature.node.kubevirt.io/sse4.2=true cpu-feature.node.kubevirt.io/ssse3=true cpu-model.node.kubevirt.io/Broadwell=true cpu-model.node.kubevirt.io/Haswell=true cpu-vendor.node.kubevirt.io/Intel=true kubernetes.io/arch=amd64 kubernetes.io/hostname=dtn-gpu2.kreonet.net kubernetes.io/os=linux mtu=9000 nautilus.io/network=40000 netbox.io/site=kreonet nvidia.com/cuda.driver-version.full=580.126.09 nvidia.com/cuda.driver-version.major=580 nvidia.com/cuda.driver-version.minor=126 nvidia.com/cuda.driver-version.revision=09 nvidia.com/cuda.driver.major=580 nvidia.com/cuda.driver.minor=126 nvidia.com/cuda.driver.rev=09 nvidia.com/cuda.runtime-version.full=13.0 nvidia.com/cuda.runtime-version.major=13 nvidia.com/cuda.runtime-version.minor=0 nvidia.com/cuda.runtime.major=13 nvidia.com/cuda.runtime.minor=0 nvidia.com/gpu.compute.major=6 nvidia.com/gpu.compute.minor=1 nvidia.com/gpu.count=5 nvidia.com/gpu.family=pascal nvidia.com/gpu.memory=12288 nvidia.com/gpu.present=true nvidia.com/gpu.product=NVIDIA-TITAN-Xp topology.kubernetes.io/region=pacific topology.kubernetes.io/zone=korea ```
Below, you can list nodes that have a certain type of node capabilities. For example, you can see which nodes provide which GPU types: ``` kubectl get nodes -L nvidia.com/gpu.product ``` This shows a cluster-wide view of GPU availability without inspecting any individual node. You can also view other commonly useful labels, such as network speed: ``` kubectl get node -l 'nautilus.io/network=100000' ``` Or combine selectors and label output: ``` kubectl get node -l 'nvidia.com/gpu.product!=NVIDIA-GeForce-GTX-1080' -L nvidia.com/gpu.product ``` ❗ Many of these queries are also available through the Nautilus portal. You can visit the [Resources](https://nrp.ai/viz/resources) page to view a live table of nodes and their features. ## Validating requirements Before adding scheduling constraints to your pod `yaml`, it’s good practice to verify that the requested resources exist somewhere in the cluster. For example, to check whether a specific GPU type is available: ``` kubectl get node -l 'nvidia.com/gpu.product=NVIDIA-GeForce-RTX-3090' ``` ❓ Did you get any results? Here we check for nodes with 100 Gbps networking: ``` kubectl get node -l 'nautilus.io/network=100000' ``` ❓ Did you get any results? 💡 Even though you cannot inspect nodes directly, the scheduler has full visibility and will match your pod requirements against all eligible nodes automatically. ## Requirements in pods You have already used resource requirements in pods. Here is a simple example: ```yaml apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: mypod image: rocker/cuda resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: ["sh", "-c", "sleep infinity"] ``` The resource requests and limits are intentionally small, making it very likely that the pod will start. ### Requesting a GPU Now let’s add a GPU requirement. ❗ Note: You cannot request a fraction of a GPU. Requests and limits must match. ```yaml apiVersion: v1 kind: Pod metadata: name: test-gpupod spec: containers: - name: mypod image: rocker/cuda resources: limits: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 requests: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 command: ["sh", "-c", "sleep infinity"] ``` Once the pod starts, log into it and verify the GPU: ``` kubectl exec test-gpupod -it -- /bin/bash ``` Inside the container, run: ``` nvidia-smi ``` ❗ Remember to delete old pods when you are done. ## Requesting a specific GPU type To require a specific GPU model, use nodeAffinity: ```yaml apiVersion: v1 kind: Pod metadata: name: test-gpupod spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: nvidia.com/gpu.product operator: In values: - NVIDIA-GeForce-RTX-3090 containers: - name: mypod image: rocker/cuda resources: limits: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 requests: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 command: ["sh", "-c", "sleep infinity"] ``` If the pod starts successfully, confirm the GPU type using nvidia-smi. ## Preferences in pods Sometimes you would prefer a resource but do not want to require it. This can be expressed using preferredDuringSchedulingIgnoredDuringExecution: ```yaml apiVersion: v1 kind: Pod metadata: name: test-gpupod spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: nvidia.com/gpu.product operator: In values: - NVIDIA-GeForce-RTX-2080-Ti - Tesla-V100-SXM2-32GB containers: - name: mypod image: rocker/cuda resources: limits: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 requests: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 command: ["sh", "-c", "sleep infinity"] ``` Check where the pod landed and which GPU you received. ## Using geographical topology Nautilus nodes are distributed globally. You can select nodes closer to your data or collaborators using topology labels. View available regions and zones: ``` kubectl get nodes -L topology.kubernetes.io/zone,topology.kubernetes.io/region ``` Run a pod in Korea: ```yaml apiVersion: v1 kind: Pod metadata: name: test-geo spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - korea containers: - name: mypod image: alpine resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: - sh - -c - | apk add curl; curl ipinfo.io ``` Check the logs: ``` kubectl logs test-geo ``` ## Optional: Reserved nodes and taints Some nodes are restricted and require explicit tolerations. You can view node taints at: https://nrp.ai/viz/resources/. :::caution Tolerating a taint you're not explicitly allowed to tolerator by the cluster admins will result in being banned. ::: If a pod cannot be scheduled, inspect events: ``` kubectl get events --sort-by=.metadata.creationTimestamp ``` You may see a NoSchedule taint. To tolerate it: ```yaml tolerations: - key: nautilus.io/reservations operator: Equals value: "name" effect: NoSchedule ``` --- # Running Batch Jobs - URL: https://nrp.ai/documentation/userdocs/tutorial/jobs/ - Section: User Guide > Tutorials - Description: Package a finite piece of computation as a Kubernetes batch job and let it run unattended. The Nautilus Cluster is designed specifically to support high-throughput batch jobs. In Kubernetes, a batch job is a type of workload designed to run a finite number of tasks to completion, as opposed to continuously running or long-lived services. Batch jobs are ideal for executing tasks such as data processing, data analysis, batch data updates, backups, or any other task that needs to be performed periodically or on-demand. A batch job (or simply, a job) is a daemon which watches your pod and makes sure it exited with exit status 0. If it did not for any reason, it will be restarted up to `backoffLimit` number of times. :::tip[What is the difference between a Job and a Pod?] A Job is a higher-level abstraction that manages a Pod. A Job will ensure that the Pod runs to completion, and can be scaled up or down as needed. A Pod is a group of one or more containers, with shared storage and network resources. ::: ## Prerequisites This section builds on skills from both the [Quickstart](https://nrp.ai/documentation/userdocs/start/getting-started/) and the tutorial on [Basic Kubernetes](https://nrp.ai/documentation/userdocs/tutorial/basic). ## Learning Objectives 1. You will learn how to create a simple job that will execute a command, then run to completion. 2. You will have a preliminary understanding of job states, such as "Completed" or "Error". 3. You will understand how to set limits to jobs :::note Since jobs in Nautilus are not limited in runtime, you can only run jobs with meaningful `command` field. Running in manual mode (`sleep infinity` `command` and manual start of computation) is prohibited. ::: Let's run a simple job and get it's result. Create a a file called `job.yaml` file and submit ito the cluster: ```yaml apiVersion: batch/v1 kind: Job metadata: name: pi spec: template: spec: containers: - name: pi image: perl command: ['perl', '-Mbignum=bpi', '-wle', 'print bpi(2000)'] resources: limits: memory: 200Mi cpu: 1 requests: memory: 50Mi cpu: 50m restartPolicy: Never backoffLimit: 4 ``` Explore what's running: ``` kubectl get jobs kubectl get pods ``` When job is finished, your pod will stay in Completed state, and Job will have COMPLETIONS field 1/1. For long jobs, the pods can have Error, Evicted, and other states until they finish properly or backoffLimit is exhausted. :::tip[How would you diagnose and fix a Job that exited with the code "error"?] You could view the logs of the job with the logs command: ```bash kubectl logs pi- ``` ::: :::note Learn more about Job states by visiting the Kubernetes documentation about [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/). There are many features of Jobs that can be utilized to run your application at-scale. ::: Our job did not use any storage and output the result to STDOUT, which can be seen as our pod logs: ``` kubectl logs pi- ``` The pod and job will remain for you to come and look at for `ttlSecondsAfterFinished=604800` seconds (1 week) by default, and you can adjust this value in your job definition if desired. You can use the [more advanced example](https://nrp.ai/documentation/userdocs/running/jobs) when ready. ## The end Please make sure you did not leave any pods and jobs behind. --- # Docker Images - URL: https://nrp.ai/documentation/userdocs/tutorial/images/ - Section: User Guide > Tutorials - Description: Find, pull, and build the container images your Nautilus workloads run on. Images are fundamental components in Nautilus as they package the application code, dependencies, and runtime environment required to run containers within Kubernetes clusters. Images serve as the building blocks for containerized applications, allowing developers to deploy their software in a consistent and portable manner. ## Learning Objectives 1. By reviewing these materials, you will have a basic understanding of how to locate standard software packages to run on Nautilus. 2. You will have a basic understanding of how to modify those packages using Docker. ## Prerequisites In order to complete this tutorial, you should have the following: 1. An account on [Docker Hub](https://hub.docker.com/signup). 2. [Docker Desktop](https://www.docker.com/get-started/) installed on your local machine. 3. An account on [Nautilus GitLab repository](https://gitlab.nrp-nautilus.io/users/sign_up). 4. You should be familiar with the concept of software container images, Kubernetes and have completed the basic Kubernetes tutorials. ## Finding container images and building your own Multiple pre-built publicly available container images already exist on [Docker Hub](https://hub.docker.com). If there's one that fits your needs, you can use one directly from there. You will need to sign up for an account, and it would be helpful for you to install Docker Desktop on your local machine. If there's an image that you want to use, but also need a couple more libraries added to it, you can extend the image by building your own based on remote one. First, create an account in our [GitLab](https://gitlab.nrp-nautilus.io) instance to simplify building and storing the containers. ## Extending an existing image Find the image you need. Let's say we want to extend the [python:3](https://hub.docker.com/_/python) image by adding a jupyter package to it, and also installing the vim text editor. Follow the [GitLab development guide](https://nrp.ai/documentation/userdocs/development/gitlab/) (make it public to simplify accessing your image, or keep it private and pull it with a deploy token as described in [Private Repos](https://nrp.ai/documentation/userdocs/development/private-repos/)), and also add this `Dockerfile` to the project before committing your changes: ``` FROM python:3 RUN pip install jupyter && \ apt-get update && \ apt-get install -y vim ``` Use the [basic section](https://nrp.ai/documentation/userdocs/tutorial/basic) to run the pod with your new image. Exec into the pod and check jupyter and vim are installed: ``` root@test-pod:/# vim -h | head -n 1 VIM - Vi IMproved 8.1 (2018 May 18, compiled Jun 15 2019 16:41:15) root@test-pod:/# jupyter usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [subcommand] jupyter: error: one of the arguments --version subcommand --config-dir --data-dir --runtime-dir --paths is required ``` ## Changing an existing image Sometimes you see an image which almost work for you, but you'd like to change versions of the software of the software installed in it. Some images provide the source of their Dockerfile, which is usually well written and you can use it as is, but modify the versions. For example, we take the [mltshp/mltshp-web](http://hub.docker.com/r/mltshp/mltshp-web) image, and go to it's [Dockerfile](https://hub.docker.com/r/mltshp/mltshp-web/dockerfile). Then we copy it's contents and use the above steps to build our own image with needed changes. ## The end Please make sure you deleted the project in `Settings -> General -> Advanced` when done. --- # Storage - URL: https://nrp.ai/documentation/userdocs/tutorial/storage/ - Section: User Guide > Tutorials - Description: Request and attach persistent storage to your workloads with Persistent Volume Claims. In Kubernetes, a Persistent Volume Claim (PVC) is a resource that allows a user to request storage from a storage class defined in the cluster. StorageClasses enable the cluster to abstract the details of storage provisioning and management, allowing users to request storage without needing to know the specifics of the underlying infrastructure. ## Prerequisites This section builds on skills from the tutorial on [Basic Kubernetes](https://nrp.ai/documentation/userdocs/tutorial/basic). You will need a basic understanding of Kubernetes concepts such as Pods, Persistent Volume Claims (PVCs), and Persistent Volumes (PVs). ## Learning objectives 1. You will have a basic understanding of storage types in Kubernetes. 2. By completing this lesson, you will understand how to request a persistent volume claim (PVC). 3. You will understand how to connect your PVC to another pod and make it available to your software container. ## Storage Types In a Kubernetes cluster, there are several types of storage options available to manage data persistence for applications and services: - **Local Storage**: Kubernetes allows pods to use storage directly attached to the node where they are scheduled. This is typically in the form of local disks. While local storage is fast, it is not portable across nodes and will be subject to data loss if the node fails. - **Persistent Volumes** (PV): PVs are cluster-wide storage resources provisioned by the (cluster) administrator. They are not bound to any particular pod, and pods can claim them using Persistent Volume Claims (PVCs). In Nautilus, PVs are created dynamically when a Persistent Volume Claim is made. - **Persistent Volume Claim** (PVC): PVCs are requests for storage by applications. They are used by developers to request specific storage resources (size, access mode, etc.) without needing to know the underlying storage implementation. - **Object Storage**: Kubernetes can also integrate with object storage systems like Amazon S3, Google Cloud Storage, and others using plugins or external solutions like MinIO. These provide scalable and durable storage for various types of data. There are other types of storage options in other Kubernetes clusters, but they are not implemented in Nautilus. ## Create an emptyDir In Kubernetes, an `emptyDir` is a type of volume that is initially empty and created when a Pod is assigned to a node. It's intended to be used as temporary storage within a pod. An `emptyDir` volume exists as long as the Pod that uses it is running on a node. When the Pod is removed from the node for any reason, the data in the emptyDir is deleted permanently. > **On Nautilus:** Pods that write more than **50Gi** of ephemeral scratch data (for example, to an `emptyDir`) can be evicted. When you use an `emptyDir` scratch volume, set `resources.requests.ephemeral-storage` (and optionally `resources.limits.ephemeral-storage`) in your container spec to cover the largest scratch size you expect. See the [Local Scratch](https://nrp.ai/documentation/userdocs/storage/local) and [High I/O Jobs](https://nrp.ai/documentation/userdocs/running/io-jobs) guides for complete examples of how to request `ephemeral-storage` together with `emptyDir`. Let’s explore the `emptyDir`by creating a simple example. You can copy-and-paste the lines below into a new file called `strg1.yaml`. ```yaml title="strg1.yaml" apiVersion: apps/v1 kind: Deployment metadata: name: test-storage labels: k8s-app: test-storage spec: replicas: 1 selector: matchLabels: k8s-app: test-storage template: metadata: labels: k8s-app: test-storage spec: containers: - name: mypod image: alpine resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: ['sh', '-c', 'apk add dumb-init && dumb-init -- sleep 100000'] volumeMounts: - name: mydata mountPath: /mnt/myscratch volumes: - name: mydata emptyDir: {} ``` :question: Examine the sample `yaml`code above. Why do you think we specified a "deployment" instead of a simple pod? Are there other things that are different about this deployment? ❗ Hint: examine the `image` we are using? :question: What image is it? Would we expect this image to behave like other images? ### Start the deployment Just like other processes, we can start our deployment by using the `create` command and specifying a file, like so: `kubectl create -f strg1.yaml` Now, try logging into the Pod created by the deployment. You will have to discover its name by querying the cluster with `kubectl get pods -o wide`. :question: Were you able to log in? If you used the command: `kubectl exec -it test-storage- -- /bin/bash` :question: What was the outcome? :question: Were you able to log into the pod? ❗ In fact, the `yaml` file is using the Linux distribution known as "Alpine". The Alpine Linux distribution is a lightweight and security-oriented Linux distribution commonly used in containerized environments, including Kubernetes. Alpine Linux is known for its minimalistic design, small footprint, and focus on security. It provides a simple and efficient base for containerized applications, offering a smaller attack surface and reduced resource usage compared to other Linux distributions. But instead of the full-featured `bash`shell (aka Command Line Interpreter or CLI), Alpine uses a lightweight version called `ash` (short for Almquist Shell). It aims to provide essential shell functionalities while keeping its codebase small and efficient. It lacks some of the advanced features found in Bash but offers POSIX compliance and basic scripting capabilities. For Alpine, we need to use a different interpreter, so use this command instead: `kubectl exec -it test-storage- -- /bin/ash` to log into the Pod. Once you are inside the Pod, you can create a directory, such as `mkdir /mnt/myscratch/` then store some files in it (hint: you can create them on the fly, using the `cat` command to redirect the standard input). Also put some files in some other (unrelated) directories, if you wish. Now, while still logged into the Pod, kill the container using the command `kill 1`. Since this is a deployment, we'd expect the container to respawn, so we can just wait for a new one to be created, then log back in. :question: What happened to files? You can now delete the deployment. ## Creating a Persistent Volume Claim In addition to the computing cluster we've been exploring, Nautilus also has a distributed storage system (a Ceph Storage Cluster). This storage cluster provides persistent storage for Nautilus. Integrating a Ceph storage cluster with a Kubernetes cluster offers several advantages for managing storage in containerized environments, such as scaleability, high availability, fault tolerance, dynamic provisioning, performance and data mobility for containerized applications. To get storage, we need to create an abstraction called `PersistentVolumeClaim`. By doing so, we "claim" some storage space and a "Persistent Volume" is created dynamically. PVCs are scoped to a particular namespace in Kubernetes. This means that PVCs created within one namespace are not directly accessible or visible to other namespaces by default. Create the file: ```yaml title="pvc.yaml" apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-vol spec: storageClassName: rook-ceph-block accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` We're creating a 1GB volume and formatting it with XFS. Look at it's status with `kubectl get pvc -o wide`. The `STATUS` field should be equals to `Bound` - this indicates successful allocation. Now we can attach it to our pod. Create one called pvc-pod.yaml: ```yaml apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: mypod image: ubuntu:latest command: ['sh', '-c', 'sleep infinity'] resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m volumeMounts: - mountPath: /examplevol name: examplevol volumes: - name: examplevol persistentVolumeClaim: claimName: test-vol ``` In volumes section we're attaching the requested persistent volume to the pod (by its name!), and in volumeMounts we're mounting the attached volume to the container in specified folder. ## Exploring storageClasses Attaching persistent storage is usually done based on storage class. You can explore the different storage classes by reading the [documentation](https://nrp.ai/documentation/userdocs/storage/intro). Not all storage classes are available to you as a User or even as a namespace administrator. Note that the one we used is the default - it will be used if you define none. ❗ Not all Linux distributions share the same functionalities, though many of the basics may be the same (e.g. POSIX-compliance). It's important to choose the right distribution based on your needs. It's also important to remember to balance your requests against your actual needs, keeping in mind that optimizing system resource requirements is important when large numbers of tasks are executed concurrently. Minimizing system resource requirements improves scalability, speed, reliability and stability. ❗ Remember that you can choose compute nodes location closer to your preferred storage class as described in the [scheduling tutorial](https://nrp.ai/documentation/userdocs/tutorial/scheduling/#using-geographical-topology). ## Cleaning up After you've deleted all the pods and deployments, delete the volume claim: `kubectl delete pvc test-vol` Please make sure you did not leave any running pods, deployments, volumes. ## End --- # Debugging - URL: https://nrp.ai/documentation/userdocs/tutorial/debugging/ - Section: User Guide > Tutorials - Description: A step-by-step method for finding out why a workload is not doing what you expected. No one writes perfect code or executes scripts without failures or errors. Understanding how to diagnose and fix these roadblocks is an important skill. Because Nautilus is highly complex and Kubernetes requires additional skills, this page is written to help you understand some basic steps you can use for debugging. Diagnosing and fixing errors in Kubernetes differs from troubleshooting issues on a physical machine in several key ways due to the distributed and dynamic nature of containerized environments. Here are some notable differences: - **Abstraction Layer**: Kubernetes abstracts away many underlying details of the infrastructure, such as physical servers, networking configurations, and storage devices. This abstraction simplifies management but can make it more challenging to diagnose low-level hardware issues compared to troubleshooting on a physical machine. - **Distributed Environment**: Kubernetes operates in a distributed environment with multiple nodes, each running multiple containers. Errors may stem from interactions between various components, including containers, pods, nodes, and the Kubernetes control plane. Diagnosing and fixing issues often involve understanding the interactions and dependencies between these distributed components. - **Dynamic Resource Allocation**: Kubernetes dynamically schedules and manages resources across the cluster, automatically distributing workloads based on resource availability and workload requirements. Diagnosing errors may involve understanding how resources are allocated and ensuring proper resource utilization across the cluster. - **Containerized Workloads**: In Kubernetes, applications run inside containers, which are isolated environments with their own filesystem, dependencies, and runtime environments. Errors may occur due to issues specific to containerized environments, such as misconfigured container images, resource constraints, networking issues, or container runtime errors. - **Declarative Configuration**: Kubernetes provides built-in features for container orchestration, including scheduling, scaling, health checks, rolling updates, and service discovery. Diagnosing errors may involve understanding how these orchestration features interact with application workloads and identifying any misconfigurations or errors in the orchestration settings. ## Step-by-step Diagnosis of errors in Kubernetes Diagnosing and fixing a Kubernetes Job that is in an "error" state involves several steps to identify the root cause of the problem and take appropriate corrective actions. Here's a general approach you can follow: 1. **Inspect Job events**: Use the `kubectl describe` command to inspect the details of the job and its associated pods. Look for any events, errors, or warnings that may provide clues about what went wrong. 2. **Check Pod Logs**: Use the `kubectl logs ` command to retrieve logs from the pods associated with the job. Look for any error messages, stack traces, or other indicators of problems that may have occurred during pod execution. 3. **Review Container Images**: Verify that the container images specified in the job's pod template are accessible and properly configured. Ensure that the images contain the necessary application code, dependencies, and configurations required for job execution. 4. **Inspect Resource Constraints**: Check if resource constraints such as CPU and memory limits are properly configured for the job and its pods. Insufficient resources may lead to pod failures and errors. 5. **Review Job Configuration**: Review the configuration of the job, including parameters such as command, arguments, environment variables, volume mounts, and resource requests/limits. Ensure that the job configuration is correct and matches the intended behavior. 6. **Check Cluster Resource Availability**: Verify that there are sufficient cluster resources (CPU, memory, and GPU, if applicable) available to execute the job. Resource constraints or contention may prevent the job from starting or completing successfully. 7. **Monitor External Dependencies**: If the job interacts with external dependencies or services, monitor the status and availability of these external components. Errors or failures in external dependencies may impact job execution. 8. **Retry or Resubmit Job**: If the error is transient or due to temporary issues, consider retrying or resubmitting the job. Kubernetes allows you to delete and recreate failed jobs, which may resolve the issue if it was caused by a transient failure. ## Implement Error Logging Error handling mechanisms such as error logging can be implemented within Kubernetes Pods using application-level logging libraries or by redirecting container output to standard output (stdout) and standard error (stderr). Here's an example of a YAML file that demonstrates error logging using Kubernetes native mechanisms: ```yaml apiVersion: v1 kind: Pod metadata: name: error-handling-pod spec: containers: - name: my-container image: my-container-image:latest command: ['sh', '-c', "echo 'Starting application'; ./my-application 2>&1 | tee /var/log/my-application.log"] resources: requests: memory: '50Mi' cpu: '100m' limits: memory: '50Mi' cpu: '100m' volumeMounts: - name: logs-volume mountPath: /var/log volumes: - name: logs-volume emptyDir: {} ``` In this example, you are redirecting the standard error output that exists in `/var/log` and writing it out to an empty directory called "logs-volume" which both exist only inside the pod. If the pod vanishes, so will this directory. Instead, you can write these messages to a `persistent volume claim` (PVC) that you created for this purpose. In this way, the messages will be stored even if the pod goes away. See the section on [Storage](userdocs/tutorial/storage) for detailed information on creating and using PVCs. --- # Container images on CVMFS - URL: https://nrp.ai/documentation/userdocs/tutorial/nrp-software/ - Section: User Guide > Tutorials - Description: Use the NRP software repository on CVMFS for reproducible, shareable command-line environments. [CVMFS](https://cernvm.cern.ch/fs/) is a distributed filesystem that allows you to mount software repositories and datasets on your device. NRP hosts an application repository `/cvmfs/nrp-software.opensciencegrid.org` to give users access to a reproducible, ready-to-use environment to share and use data through CVMFS. In this tutorial, we'll go through steps distributing and using container images to the CVMFS repo hosted by NRP. **Access to container images distributed to CVMFS is available globally. For example, the CVMFS data repository is also availabale on the [OSG OSPool](https://osg-htc.org/services/ospool/), not just limited to the NRP environment. And other repositories in CVMFS can also be accessed from NRP environment.** ## Learning Objectives 1. How to create your own data images. 2. How to distribute images into CVMFS. 3. How to use the images distributed in CVMFS ## Prerequisites In order to complete this tutorial, you should have gone through the [Quickstart](https://nrp.ai/documentation/userdocs/start/getting-started/), and finished these tutorials: 1. [Basic Kubernetes](https://nrp.ai/documentation/userdocs/tutorial/basic) 2. [Storage](https://nrp.ai/documentation/userdocs/tutorial/storage) 3. [Docker Images](https://nrp.ai/documentation/userdocs/tutorial/images) You will also need knowlege regarding [Creating a pull request from a fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) on GitHub. ## Creating data container images Here we focus on how to create a customized data container image. We suppose that you have setup your enviroment according to the [Docker Images](https://nrp.ai/documentation/userdocs/tutorial/images) tutorial. ### 1. Put installation instructions in a `Dockerfile` A `Dockerfile` is a plain text file with keywords and commands that can be used to create a new container image. Here is an example to build an image based on [python:3](https://hub.docker.com/_/python) with the `jupyter` package, and a data file called `data.txt`. ``` FROM python:3 RUN pip install jupyter ADD data.txt /data.txt ``` - `FROM`, indicates which container image we’re starting with. We use [python:3](https://hub.docker.com/_/python) as the base image. - `RUN`, indicates installation commands we want to run while building the image. Here we use `pip` to install the `jupyter` package. - `ADD`, indicates the local or remote files or directories to be included in the image. Here we include a local file `data.txt` in the image. ### 2. Build the image Run the following command to build the image in the same directory with the `Dockerfile` and `data.txt` files: ```sh docker build . -t my-data-container:latest ``` ### 3. Push the image to a container registry In Step 2, we built a local image. If you have an account on [Docker Hub](https://docs.docker.com/docker-hub/), you can run the following commands to tag the image and push it there: ```sh docker image tag my-data-container:latest DOCKER_USERNAME/my-data-container:latest docker push DOCKER_USERNAME/my-data-container:latest ``` You may also use the GitLab container registry provided by NRP. For details, please refer to [Building in GitLab](https://nrp.ai/documentation/userdocs/development/gitlab) ## Distributing container images on CVMFS Image distribution on CVMFS works with unpacked layers or image root file systems. We host a CVMFS repo to synchronize and distribute container images. Any image publically available in Docker and other registries can be included for automatic syncing into the CVMFS repository. The result is an unpacked image under `/cvmfs/nrp-software.opensciencegrid.org`. To get your images included, please either create a git pull request against `images.txt` in the [cvmfs-oci](https://github.com/nrp-nautilus/cvmfs-oci) repository, or [contact NRP admins](https://nrp.ai/contact) and we can help you. The `images.txt` file is a list of container images, one image per line, in the format of `[registry_host/][namespace/]repository_name[:tag]`. For examaple: ``` gitlab-registry.nrp-nautilus.io/nrp/scientific-images/python:latest ``` When this line is added to the image.txt and the pull request is merged, the image will be unpacked into CVMFS at `/cvmfs/gitlab-registry.nrp-nautilus.io/nrp/scientific-images/python:latest`. If new versions of images have been pushed to the registry, they will be detected and updated in CVMFS accordingly. ## Accessing unpacked container images ### Creating a PVC volume with the spec `storageClassName: cvmfs` To access images distributed in CVMFS, you need to attach the CVMFS volume which can mount all repos. First, create the PVC (taken from [https://github.com/cvmfs-contrib/cvmfs-csi/tree/master/example](https://github.com/cvmfs-contrib/cvmfs-csi/tree/master/example) ): ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cvmfs spec: accessModes: - ReadOnlyMany resources: requests: # Volume size value has no effect and is ignored # by the driver, but must be non-zero. storage: 1 storageClassName: cvmfs ``` ### Using the `cvmfs` PVC in pods #### Option 1, mount the entire CVMFS Create a pod with the following yaml file: ```yaml apiVersion: v1 kind: Pod metadata: name: cvmfs-all-repos spec: containers: - name: idle image: busybox imagePullPolicy: IfNotPresent command: ['/bin/sh', '-c', 'trap : TERM INT; (while true; do sleep 1000; done) & wait'] volumeMounts: - name: my-cvmfs mountPath: /my-cvmfs # CVMFS automount volumes must be mounted with HostToContainer mount propagation. mountPropagation: HostToContainer volumes: - name: my-cvmfs persistentVolumeClaim: claimName: cvmfs ``` In this example, the `nrp-software.opensciencegrid.org` repo is accessible at `/my-cvmfs/nrp-software.opensciencegrid.org` in the pod. Notice that repo is mounted unless the mount point is accessed. #### Option 2, mount the `nrp-software.opensciencegrid.org` repo specifically If you need to mount the `nrp-software.opensciencegrid.org` repo specifically, add the `subPath` key to the pod's `volumeMounts` section: ```yaml volumeMounts: - name: my-cvmfs # It is possible to mount a single CVMFS repository by specifying subPath. subPath: nrp-software.opensciencegrid.org mountPath: /my-nrp-software-cvmfs mountPropagation: HostToContainer ``` In this example, the repo is accessible at `/my-nrp-software-cvmfs`, but other repos in CVMFS are not available in the pod. ### Mount the `cvmfs` PVC in customized JupyterHub deployment If you have a [customized JupyterHub deployment](https://nrp.ai/documentation/userdocs/jupyter/jupyterhub), you can make CVMFS available in user spawned instances. Suppose a `cvmfs` PVC has been created in the namespace where the Jupyterhub is depoloyed, and you want to mount the CVMFS repo `nrp-software.openscience.org` at /nrp-software in every user's pod, you can insert the following example into the JupyterHub's value template (e.g. config.yaml): ```yaml singleuser: storage: extraVolumes: - name: nrp-software persistentVolumeClaim: claimName: cvmfs extraVolumeMounts: - name: nrp-software mountPath: /nrp-software subPath: nrp-software.opensciencegrid.org mountPropagation: HostToContainer ``` And then update the helm chart by command `helm upgrade --cleanup-on-fail --install jhub jupyterhub/jupyterhub --namespace --version= --values config.yaml`. When the JupyterHub is deployed, the images in `nrp-software.opensciencegrid.org` CVMFS repo is mounted as /nrp-software when users spawn new pods. --- # GPU Pods - URL: https://nrp.ai/documentation/userdocs/running/gpu-pods/ - Section: User Guide > Running > Run your first job - Description: Request GPUs in a pod spec, including high-memory and MIG GPU types, and clean up when done. :::note In this section you will request GPUs. Make sure you don't waste those and delete your pods when not using the GPUs. ::: :::caution Some specific high-memory GPUs require the gpu type specified in the container `resource` requests and limits. See [Requesting special GPUs](#requesting-special-gpus) ::: ## Running GPU pods Use this definition to create your own pod and deploy it to kubernetes: ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod-example spec: containers: - name: gpu-container image: tensorflow/tensorflow:latest-gpu command: ['sleep', 'infinity'] resources: limits: nvidia.com/gpu: 1 requests: nvidia.com/gpu: 1 ``` This example requests 1 GPU device. You can have up to 8 per node if you're [using jobs](https://nrp.ai/documentation/userdocs/running/jobs/), and up to 2 for pods. If you request GPU devices in your pod, kubernetes will auto schedule your pod to the appropriate node. There's no need to specify the location manually. :::caution[Delete your pod when your computation is done] A GPU stays reserved for as long as your pod exists, so an idle pod blocks it for everyone else. Consider using [Jobs](https://nrp.ai/documentation/userdocs/running/jobs/) **with actual script instead of `sleep`** whenever possible to ensure your pod is not wasting GPU time. If you have never used Kubernetes before, see the [tutorial](https://nrp.ai/documentation/userdocs/tutorial/introduction/). ::: ## Requesting special GPUs Certain kinds of GPUs are advertised on nodes as a special resource, f.e. "nvidia.com/rtx-8000". You have to request that resource instead of the "nvidia.com/gpu" one. The current list is: | GPU Type | Resource | | ----------------------------- | -------------------- | | NVIDIA A40 | nvidia.com/a40 | | NVIDIA A100 | nvidia.com/a100 | | NVIDIA RTX A6000 | nvidia.com/rtxa6000 | | NVIDIA Quadro RTX 8000 | nvidia.com/rtx8000 | | NVIDIA H100 | nvidia.com/h100 | | NVIDIA H200 | nvidia.com/h200 | | NVIDIA RTX PRO 6000 Blackwell | nvidia.com/rtx6000bw | | NVIDIA GH200 Grace Hopper | nvidia.com/gh200 | | NVIDIA A100 MIG 1g.10gb | nvidia.com/mig-small | > **Note:** NVIDIA RTX PRO 6000 Blackwell nodes are reserved for exclusive use and are not generally available at this time. :::caution[Access policy for A100 / H100 / H200 / GH200] These four GPU types are gated by a per-namespace ResourceQuota. By default every namespace has a quota of **zero** for each of them. To run a pod on one of them you have one of these paths: - **NVIDIA A100** — user-requestable. Submit the [A100 access request](https://nrp.ai/reservations) and an admin will raise your namespace's a100 quota. - **NVIDIA H100, H200, and GH200** — **not user-requestable.** These are reserved for the groups that contributed the hardware, plus LLM workloads consuming spare cycles. There is no access form. - **Any user, any of the four GPUs, no reservation** — set `priorityClassName: opportunistic` (or `opportunistic2`) on your pod. The quota does not apply to opportunistic-tier pods, but they can be preempted by any other pod at any time. See [Priority Classes](https://nrp.ai/documentation/userdocs/running/priority-classes/) for details and a full example. ::: For example, modifying the above example for one of these GPUs, the new yaml would be: ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod-example spec: containers: - name: gpu-container image: tensorflow/tensorflow:latest-gpu command: ['sleep', 'infinity'] resources: limits: nvidia.com/a100: 1 requests: nvidia.com/a100: 1 ``` For Grace Hopper node make sure you're also using the image with `arm` support (`nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04` is a good one) and [tolerating the arm64 architecture](https://nrp.ai/documentation/userdocs/running/special/): ```yaml tolerations: - key: 'nautilus.io/arm64' operator: 'Exists' effect: 'NoSchedule' ``` For RTX PRO 6000 Blackwell nodes, request the dedicated resource: ```yaml resources: limits: nvidia.com/rtx6000bw: 1 requests: nvidia.com/rtx6000bw: 1 ``` ## Requesting many GPUs Since 1 and 2 GPU jobs are blocking nodes from getting 4 and 8-GPU jobs, there are some nodes reserved for those. Once you submit a job with 4 or 8 GPUs request, a controller will automatically add toleration which will allow you to use the node reserved for more GPUs. You don't need to do anything manually for that. ## Choosing GPU type **See [requesting special GPUs](#requesting-special-gpus) for special types of GPU** We have a variety of GPU flavors attached to Nautilus. This table describes the types of GPUs available for use, but is not up to date - it's better to use the actual cluster information (f.e. `kubectl get nodes -L nvidia.com/gpu.product`). Credit: [GPU types on Nautilus](https://observablehq.com/d/7c0f46855b4212e0) If you need more graphical memory, use this table or official specs to choose the type: | GPU Type | Memory Size (GB) | | -------------------------- | ---------------- | | NVIDIA GeForce GTX 1070 | 8 | | NVIDIA GeForce GTX 1080 | 8 | | NVIDIA Quadro M4000 | 8 | | NVIDIA A100 MIG 2g.10gb | 10 | | NVIDIA GeForce GTX 1080 Ti | 12 | | NVIDIA GeForce RTX 2080 Ti | 12 | | NVIDIA TITAN Xp | 12 | | NVIDIA Tesla T4 | 16 | | NVIDIA A10 | 24 | | NVIDIA GeForce RTX 3090 | 24 | | NVIDIA GeForce RTX 4090 | 24 | | NVIDIA TITAN RTX | 24 | | NVIDIA RTX A5000 | 24 | | NVIDIA Quadro RTX 6000 | 24 | | NVIDIA Tesla V100 SXM2 | 32 | | NVIDIA A40 | 48 | | NVIDIA L40 | 48 | | NVIDIA RTX A6000 | 48 | | NVIDIA Quadro RTX 8000 | 48 | | NVIDIA A100 SXM4 | 80 | | NVIDIA H200 NVL | 141 | :::note [Not all nodes are available to all users](https://nrp.ai/documentation/userdocs/running/special/). You can consult about your available resources in [Nautilus Support](https://nrp.ai/contact) and on [resources page](https://nrp.ai/viz/resources). Labs connecting their hardware to our cluster have preferential access to all our resources. ::: :::caution For higher memory GPUs, use the [requesting special GPUs syntax](#requesting-special-gpus)! Affinity allows you to further refine the GPU type or choose the GPU type for generic GPUs. ::: To use a **specific type of GPU**, add the affinity definition to you pod yaml file. The example below specifies _1080Ti_ GPU: ```yaml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: nvidia.com/gpu.product operator: In values: - NVIDIA-GeForce-GTX-1080-Ti ``` **To make sure you did everything correctly** after you've submited the job, look at the corresponding pod yaml (`kubectl get pod ... -o yaml`) and check that resulting nodeAffinity is as expected. ## Selecting CUDA version In general the higher CUDA versions support the lower and same driver version. The nodes are labelled with the major and minor CUDA and driver versions. You can check those at the [resources page](https://nrp.ai/viz/resources) or list with this command (it will also choose only GPU nodes): ```bash kubectl get nodes -L nvidia.com/cuda.driver.major,nvidia.com/cuda.driver.minor,nvidia.com/cuda.runtime.major,nvidia.com/cuda.runtime.minor -l nvidia.com/gpu.product ``` If you're using the container image with higher CUDA version, you have to pick the nodes supporting it. Example: ```yaml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: nvidia.com/cuda.runtime.major operator: In values: - '12' - key: nvidia.com/cuda.runtime.minor operator: In values: - '2' ``` Also you can choose the driver above something if you know which one you need (this will pick drivers **above** 535): ```yaml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: nvidia.com/cuda.driver.major operator: Gt values: - '535' ``` ## Adding Shared Memory (shm) :::note[Adding Shared Memory (shm)] You can add Shared Memory (shm) to your GPU pods in YAML: ```YAML volumeMounts: - mountPath: /dev/shm name: dshm volumes: - name: dshm emptyDir: medium: Memory sizeLimit: 2Gi ``` `sizeLimit` is an optional term. Otherwise it defaults to half of the memory request. Not adding /dev/shm defaults to 64MB. ::: --- # Long Idle Pods - URL: https://nrp.ai/documentation/userdocs/running/long-idle/ - Section: User Guide > Running > Run your first job - Description: Keep an idle but occasionally-used service up past the pod lifetime limit by running it as a Deployment. ## Running an idle deployment In case you need to have an idle pod in the cluster, that might ocassionally do some computations, you have to run it as a [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). Deployments in Nautilus are limited to 2 weeks (unless the namespace is added to exceptions list and runs a permanent service). This ensures your pod will not run in the cluster forever when you don't need it and move on to other projects. Please don't run such pods as Jobs, since those are not purged by the cleaning daemon and will stay in the cluster forever if you forget to remove those. Such a deployment **can not request a GPU**. You can use the ``` command: - sleep - "100000000" ``` as the command if you just want a pure shell, and `busybox`, `centos`, `ubuntu` or any other general image you like. Follow the [guide for creating deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and add the minimal requests to it and limits that make sense, for example: ``` resources: limits: cpu: "1" memory: 10Gi requests: cpu: "10m" memory: 100Mi ``` Example of running an nginx deployment: ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: k8s-app: nginx spec: replicas: 1 selector: matchLabels: k8s-app: nginx template: metadata: labels: k8s-app: nginx spec: containers: - image: nginx name: nginx-pod resources: limits: cpu: 1 memory: 4Gi requests: cpu: 100m memory: 500Mi ``` ## Quickly stopping and starting the pod If you need a simple way to start and stop your pod without redeploying every time, you can scale down the deployment. This will leave the definition, but delete the pod. To stop the pod, scale down: ``` kubectl scale deployment deployment-name --replicas=0 ``` To start the pod, scale up: ``` kubectl scale deployment deployment-name --replicas=1 ``` --- # Monitoring - URL: https://nrp.ai/documentation/userdocs/running/monitoring/ - Section: User Guide > Running > Run your first job - Description: Watch what your workloads actually consume in Grafana and correct resource requests to match. When you run your jobs, it's your responsibility to make sure they are running as intended, without overrequesting the resources. Our [Grafana](https://grafana.nrp-nautilus.io/) page is a great resource to see what your jobs are doing. To get an idea how much resources your jobs are using, go to [namespace dashboard](https://grafana.nrp-nautilus.io/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods) and choose your namespace in the dropdown list on top. Your requests percentage for memory and CPU should be as close to 100% as possible. Also check the [GPU dashboard](https://grafana.nrp-nautilus.io/d/dRG9q0Ymz/k8s-compute-resources-namespace-gpus) for your namespace to make sure the utilization is above 40%, and ideally is close to 100%. When checking the memory utilization, make sure to use the **Memory Usage (RSS)** column. The **Memory Usage** includes the disk cache, which can grow indefinitely. You can use `time` in your yaml command to print the maximum memory utilization. The following yaml configuration will do this. Note that you will need to `apt-get install time` in your Dockerfile to use this command line tool. ``` apiVersion: batch/v1 kind: Job metadata: name: myname-poddescription spec: template: spec: containers: - command: ["stdbuf", "-i0", "-o0", "-e0", "/usr/bin/time", "-v", "bash", "-c"] args: - >- python -u myprocess.py --arg1 "a" --arg2 "b" ``` Description: - `stdbuff -i0 -o0 -e0` Unbuffers output so that `kubectl logs --follow PODNAME` writes output immediately, without unbuffering your process may appear hung when it's just line buffering the output. - `/usr/bin/time` Prints the processes time and usage statistics at the end of the run, most importantly this includes maximum memory utilization. You will find this output via `kubectl logs PODNAME`. - `bash -c` Executes the value of `args` as a single line bash command. It's useful to do this way so that the command being executed runs the same as it does from the bash command line. - `args: - >-` This is a directive to the yaml parser to concatenate multiple lines. This makes the command easy to read and write. The command can include pipes, and you can string together multiple commands with `;` or `&&` in the same way you do on the bash command line without any need for escape characters (thanks to passing the `args` string to `bash -c`). ## Tensorboard If you are training models such as neural networks, statistical models, and the like on platforms such as Python, Tensorflow, PyTorch, etc, it is common to plot real time statistics to tools such as Tensorboard. Tensorboard in particular is an excellent real time visualization tool, but requires that you launch the Tensorboard process and keep track of the log files, all of which are extra steps to deal with under cluster environments such as the NRP. You should first activate tensorboard in the pods: ``` tensorboard --logdir=${LOG-FILE} ``` The kubectl can link you local port to the specified port of the pods: ``` kubectl port-forward ${POD_NAME} ${REMOTE-PORTNUM}:${LOCAL-PORTNUM} ``` Then the website for tensorboard can be seen in **http://localhost:${LOCAL-PORTNUM}** ## Comet.ml An alternative solution is to use [http://comet.ml](https://www.comet.ml), which is free for academic users, and provides a similar set of functions as Tensorboard (plus a Baysian Hyperparameter Tuning tool). Comet.ml stores everything on their website, so there are no logs to maintain or servers to run, and this makes it an easy solution to deploy on a distributed cluster like the NRP. --- # Running Batch Jobs - URL: https://nrp.ai/documentation/userdocs/running/jobs/ - Section: User Guide > Running > Run your first job - Description: Run your computations as Kubernetes Jobs so results are kept and nothing needs babysitting. We highly recommend using [Jobs][1] for any kind of development and computations in our cluster. This will ensure you never lose your work, get the results in the most convenient way, and don't waste resources, since this method does not require any babysitting of processes from you. Once your development is done, you are immediately ready to run a large-scale stuff with no changes to the code and minimal changes in the definition, plus your changes are saved in Git. :::danger[Idle jobs can get your account banned] Since jobs in Nautilus can run forever, you can only run jobs with meaningful `command` field. Running in interactive mode (`sleep infinity` command and manual start of computation) or any command that doesn't end by itself is prohibited, and user **can be banned**. ::: Use [our tutorial](https://nrp.ai/documentation/userdocs/tutorial/jobs/) for a simple job example. ## Pulling code from GIT You can put your code to our [ GitLab](https://gitlab.nrp-nautilus.io) repository and pull it from there using the following example. To modify your code between iterations you can use the [Web IDE][2] - simply click the *Web IDE* button on your project's repository page once you are logged in on Gitlab. Make sure your repo is not private or use the [private repo example](#private-repo). ```yaml apiVersion: batch/v1 kind: Job metadata: name: myapp spec: template: spec: containers: - name: demo image: gitlab-registry.nrp-nautilus.io/prp/jupyter-stack/prp command: - "python" args: - "/opt/repo/REPONAME/my_script.py" - "arg_job_to_run" volumeMounts: - name: git-repo mountPath: /opt/repo resources: limits: memory: 6Gi cpu: "6" nvidia.com/gpu: "1" requests: memory: 4Gi cpu: "1" nvidia.com/gpu: "1" initContainers: - name: init-clone-repo image: alpine/git args: - clone - --single-branch - https://gitlab.nrp-nautilus.io/USERNAME/REPONAME - /opt/repo/REPONAME volumeMounts: - name: git-repo mountPath: /opt/repo volumes: - name: git-repo emptyDir: {} restartPolicy: Never backoffLimit: 5 ``` Two containers, *init-clone-repo* and *demo*, share the initially empty storage volume. This pod will: - start initContainer, pull your code from Git repository and put it in /opt/repo/REPONAME - then will start your main container, and execute the script that was downloaded from the git repo - when script is finished, terminate the whole pod. ## Running several bash commands You can group several commands, and use pipes, like this: ``` command: - sh - -c - "cd /home/user/my_folder && apt-get install -y wget && wget pull some_file && do something else" ``` ## Logs All stdout and stderr output from the script will be preserved and accessible by running ``` kubectl logs pod_name ``` Output from initContainer can be seen with ``` kubectl logs pod_name -c init-clone-repo ``` To see logs in real time do: ``` kubectl logs -f pod_name ``` The pod will remain in Completed state until you delete it or timeout is passed. ## Retries The backoffLimit field specifies how many times your pod will run in case the exit status of your script is not 0 or if pod was terminated for a different reason (for example a node was rebooted). It's a good idea to have it more than 0. ## Fair queueing There is no fair queue implemented on Nautilus. If you submit 1000 jobs, you block **all** other users from submitting in the cluster. To limit your submission to a fair portion of the cluster, refer to [this guide](https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/). Make sure to use a deployment and persistent storage for Redis pod. Here's [our example](https://gitlab.nrp-nautilus.io/prp/job-queue/-/blob/master/redis.yaml) ## Private repo If your repository is **private**, you should create a Gitlab Personal Access Token (see [Access Tokens][4] and [Access Tokens for command line][5]) of type *read_repository*, and put it in your namespace secret to be consumed by the pod: ``` kubectl create secret generic gitlab-secret --from-literal=user=USERNAME --from-literal=password=TOKEN ``` The pod yaml file references the secret via `user` and `password` keys (must match secret literal definitions in the above command: ```yml apiVersion: batch/v1 kind: Job metadata: name: myapp spec: template: spec: containers: - name: demo image: gitlab-registry.nrp-nautilus.io/prp/jupyter-stack/prp command: - "python" args: - "/opt/repo/REPONAME/my_script.py" - "arg_job_to_run" volumeMounts: - name: git-repo mountPath: /opt/repo resources: limits: memory: 6Gi cpu: "6" nvidia.com/gpu: "1" requests: memory: 4Gi cpu: "1" nvidia.com/gpu: "1" initContainers: - name: init-clone-repo image: alpine/git env: - name: GIT_USERNAME valueFrom: secretKeyRef: name: gitlab-secret key: user - name: GIT_PASSWORD valueFrom: secretKeyRef: name: gitlab-secret key: password args: - clone - --single-branch - https://$(GIT_USERNAME):$(GIT_PASSWORD)@gitlab.nrp-nautilus.io/USERNAME/REPONAME - /opt/repo/REPONAME volumeMounts: - name: git-repo mountPath: /opt/repo volumes: - name: git-repo emptyDir: {} restartPolicy: Never backoffLimit: 5 ``` You can use several Work Queue Brokers, like Redis or RabbitMQ, to distribute tasks once you're ready to scale out your computation. [1]: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ [2]: https://docs.gitlab.com/ce/user/project/repository/web_editor.html [3]: https://gitlab.nrp-nautilus.io [4]: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html [5]: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line --- # CPU Only Jobs - URL: https://nrp.ai/documentation/userdocs/running/cpu-only/ - Section: User Guide > Running > Run your first job - Description: Run large CPU-only jobs politely on a GPU-focused cluster using priority classes and preemption. ## Running large CPU-only jobs Nautilus is primarily used for GPU jobs. While it's possible to run large CPU-only jobs, you have to take certain measures to prevent taking over all cluster resources. You can run the jobs with lower priority and allow other jobs to preempt yours. This way you should not worry about the size of your jobs and you can use the maximum number of resources in the cluster. To do that, add the `opportunistic` priority class to your pods: ```yaml spec: priorityClassName: opportunistic ``` Another thing to do is to avoid the GPU nodes. This way you can be sure you're only using the CPU-only nodes and jobs are not preventing any GPU usage. To do this, add the node antiaffinity for GPU device to your pod: ```yaml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: feature.node.kubernetes.io/pci-10de.present operator: NotIn values: - 'true' ``` You can use a combination of 2 methods or either one. --- # Scheduling - URL: https://nrp.ai/documentation/userdocs/running/scheduling/ - Section: User Guide > Running > Advanced patterns - Description: Scheduling large jobs :::caution Current support for everything on this page is experimental. We might change it at anytime. We welcome friendly users to test this function and would like feedback. ::: :::caution[Job specs are visible to every user] Job specs (the YAML you submit) are visible to every user of the cluster, and there is currently no way to restrict that per namespace. Do not put secrets, credentials, or unpublished data in a job spec. ::: ## Introduction [Armada](http://armadaproject.io/) multi-Kubernetes cluster batch job meta-scheduler designed to handle massive-scale workloads. Built on top of Kubernetes, Armada enables organizations to distribute millions of batch jobs per day across tens of thousands of nodes spanning multiple clusters, making it an ideal solution for high-throughput computational workloads. ## Installation To start using Armada [download the client application armadactl](https://github.com/armadaproject/armada/releases). [Download Armadactl for MacOS/arm](https://github.com/armadaproject/armada/releases) Put it in your PATH folder (f.e. `~/bin/`) and make runnable (`chmod +x ~/bin/armadactl`) Save the `.armadactl.yaml` config file in your home folder: [Download .armadactl.yaml config file](https://nrp.ai/.armadactl.yaml) You're ready to use armada. ## Usage Start from listing the queues: ```bash armadactl get queues ``` It will list all queues matching the list of all namespaces in the cluster. You can submit to the ones you normally access in the cluster. Submit a job: ```yaml title="test-job.yaml" queue: jobSetId: job-set jobs: - namespace: priorityClassName: armada-default podSpec: terminationGracePeriodSeconds: 0 restartPolicy: Never containers: - name: tester image: alpine:latest command: - sh args: - -c - echo $(( (RANDOM % 60) + 10 )) resources: limits: memory: 128Mi cpu: 2 requests: memory: 128Mi cpu: 2 ``` ```bash armadactl submit test-job.yaml ``` Login to the [lookout page](https://armada-lookout.nrp-nautilus.io) and see the status of your job. Also you should see the pod running in your namespace. ### Priorities Allowed priorities in armada: |priorityClass|preemptible|value| |---|---|---| |armada-default|false|100| |armada-preemptible|true|50| |armada-high-priority|false|1000| :::caution Preempted jobs will **not** be automatically resheduled. ::: :::note Preemption is only working inside armada. Once the pods are in k8s cluster, those won't be preempted by normal pods or preempt other cluster pods. ::: ## Python client To use python client, you can get the OIDC token from the [lookout page](https://armada-lookout.nrp-nautilus.io) in web browser. After logging in, check the site storage developer tab. There will be the id_token stored for authentik. It's good for 30 minutes. Example of the python client: ```bash pip install armada-client ``` ```python import os import uuid import grpc from armada_client.client import ArmadaClient from armada_client.k8s.io.api.core.v1 import generated_pb2 as core_v1 from armada_client.k8s.io.apimachinery.pkg.api.resource import ( generated_pb2 as api_resource, ) def create_dummy_job(client: ArmadaClient): """ Create a dummy job with a single container. """ pod = core_v1.PodSpec( containers=[ core_v1.Container( name="container1", image="index.docker.io/library/ubuntu:latest", args=["sleep", "10s"], securityContext=core_v1.SecurityContext(runAsUser=1000), resources=core_v1.ResourceRequirements( requests={ "cpu": api_resource.Quantity(string="120m"), "memory": api_resource.Quantity(string="510Mi"), }, limits={ "cpu": api_resource.Quantity(string="120m"), "memory": api_resource.Quantity(string="510Mi"), }, ), ) ], ) return [client.create_job_request_item(priority=0, pod_spec=pod, namespace="your_namespace")] token = "your_id_token" HOST = "armada.nrp-nautilus.io" PORT = "50051" queue = f"your_namespace" job_set_id = f"simple-jobset-{uuid.uuid1()}" class BearerAuth(grpc.AuthMetadataPlugin): def __call__(self, context, callback): callback((("authorization", f"Bearer {token}"),), None) channel = grpc.secure_channel(f"{HOST}:{PORT}", grpc.composite_channel_credentials( grpc.ssl_channel_credentials(), grpc.metadata_call_credentials(BearerAuth()) ) ) client = ArmadaClient(channel) job_request_items = create_dummy_job(client) client.submit_jobs( queue=queue, job_set_id=job_set_id, job_request_items=job_request_items ) print("Completed Workflow") ``` --- # Opportunistic Use - URL: https://nrp.ai/documentation/userdocs/running/priority-classes/ - Section: User Guide > Running > Advanced patterns - Description: When to set priorityClassName on your pods, and the only one you usually want — opportunistic In 95% of cases you don't need to set `priorityClassName`. The only common reason to set it is to use **A100 / H100 / H200 / GH200** GPUs without a reservation, by setting `priorityClassName: opportunistic`. :::tip[Rules of thumb] - **Default workload?** Don't set `priorityClassName`. Done. - **Want a special GPU without a reservation?** Set `priorityClassName: opportunistic` — your pod bypasses the GPU quota but can be preempted at any time. - Don't set `default`, `owner`, `nice`, `batch-low`, or any `*-critical` class — admission will reject the pod. ::: ## What you're allowed to set | Priority class | Value | User pods | GPU quota | |---|---:|---|---| | *(unset — the default)* | 0 | ✅ allowed | enforced | | `armada-default` | 0 | ✅ allowed | enforced | | `owner-no-preempt` | 10 | ✅ allowed | enforced | | `opportunistic2` | −1.99×10⁹ | ✅ allowed | 🔓 bypassed | | `opportunistic` | −2×10⁹ | ✅ allowed | 🔓 bypassed | Everything else (`default`, `owner`, `nice`, `batch-low`, `system-*`, `kubevirt-*`, `*-batch-high`, etc.) is banned in user namespaces. If your tooling auto-sets one of these, remove that line — your pod will run at the unset default, which is what you want. Exemption requests for legitimate cases go through [Nautilus Support](https://nrp.ai/contact). ## Using opportunistic for special GPUs ```yaml apiVersion: v1 kind: Pod metadata: name: h200-opportunistic spec: priorityClassName: opportunistic containers: - name: gpu-container image: nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 command: ["python", "train.py"] resources: limits: nvidia.com/h200: 1 requests: nvidia.com/h200: 1 ``` The pod can be preempted by any other pod in the cluster, so checkpoint frequently or run it as a [Job](https://nrp.ai/documentation/userdocs/running/jobs/) so the controller can restart it. `opportunistic2` is the same idea but sits one step above `opportunistic` in priority — useful only if you don't want to be preempted by *other* opportunistic workloads. ## If your pod is rejected `pods "..." is forbidden: exceeded quota: high-priority-ban` or `low-priority-ban` means your `priorityClassName` is on the ban list. Drop the field or switch to one of the allowed classes above. --- # Client Scripts - URL: https://nrp.ai/documentation/userdocs/running/scripts/ - Section: User Guide > Running > Advanced patterns - Description: Authenticate scripts and CI runs against the cluster with service-account tokens instead of the portal kubeconfig. If you run a script on your own machine that connects to kubernetes (either by using kubectl or directly), you should avoid using the config file provided by the portal, since token refresh doesn't work in concurrent runs. Instead you should create a service account in your namespace and connect using the token generated by kubernetes. All further actions can only be done by a **namespace admin**. If you have a user status, ask your admin to do this. To create a service account, run: ```shell kubectl create serviceaccount ``` Replace `` with desired name. > **Note**: In the previous kubernetes versions, the above command also created a token in a secret object. > (As of 8/16/2023, the server version is v1.25.12.) > But this is no longer the case, see [here](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-an-api-token-for-a-serviceaccount). Run the following command to create a token: ```shell kubectl apply -f - <-secret annotations: kubernetes.io/service-account.name: type: kubernetes.io/service-account-token EOF ``` Retrive the token using the following commands: ```shell TOKEN=`kubectl get secret -secret -o jsonpath='{.data.token}'| base64 --decode` echo $TOKEN ``` After that COPY your current OIDC config file, and add the new user instead of the current one. We assume that `.kube` is located in the current directory, often it is the user's home directory `~`. ```shell cp .kube/config .kube/config_sa kubectl --kubeconfig=.kube/config_sa config unset users.$(kubectl --kubeconfig=.kube/config_sa config view -o jsonpath='{.users[0].name}') kubectl --kubeconfig=.kube/config_sa config set-credentials --token=$TOKEN kubectl --kubeconfig=.kube/config_sa config set-context --current --user= kubectl --kubeconfig=.kube/config_sa config view ``` > **Note**: You may need to remove more than one user if your `config` contains more than one user. Now you need to let the service account act on behalf of user. To do this, run: ```shell kubectl create rolebinding -sa --clusterrole= --serviceaccount=: ``` Namespace admins have the `admin` role, and users typically have the `edit` role, see https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles. Now check if you can list pods: ```shell kubectl --kubeconfig=.kube/config_sa get pods ``` (You can also set the environmental variable `$KUBECONFIG` with the location of your config file for the script) To delete the service account, run: ```shell kubectl delete rolebinding -sa kubectl delete serviceaccount rm .kube/config_sa ``` > **Note**: The secret will be automatically removed by running `kubectl delete serviceaccount`. [Another guide how to do this](https://docs.cloud.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengaddingserviceaccttoken.htm) --- # Special Use - URL: https://nrp.ai/documentation/userdocs/running/special/ - Section: User Guide > Running > Advanced patterns - Description: Special Use Taints and Tolerations Our cluster combines various hardware resources from multiple universities and other organizations. :::caution By default you can only use the nodes **having NO Taints** (see the [resources page](https://nrp.ai/viz/resources) of the portal). ::: ## All Taints Please use caution when applying tolerations and only tolerate taints for which you have explicit authorization from cluster administrators. Tolerating the wrong taints may cause your workloads to land on unintended or restricted nodes, leading to failures or policy violations. Here is the taint system and their descriptions. To run on a node with a taint, you need to [use the node toleration in your pod](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). Users **may only tolerate values they are authorized for by cluster admins**. | Taint Key | Description | |--------------------------|------------------------------------------------------------------------------------------------------------------------------| | **nautilus.io/reservation** | For user-facing reservations. Users can only tolerate a value here if they are part of an approved group. If unsure, toleration is not allowed. | | **nautilus.io/hardware** | For special hardware nodes that users should not land on by default but are permitted to tolerate if needed. | | **nautilus.io/system** | For system services or any infrastructure nodes that users should never schedule onto. Toleration is not permitted. | | **nautilus.io/issue** | For nodes with temporary issues that should not accept user workloads. Numeric values indicate GitLab issues, strings indicate other issues. Users cannot tolerate these. | --- ### Taint name mappings from old to new system | Old Taint | New Taint | |-------------------------------|-------------------------------------------| | nautilus.io/5g=true | nautilus.io/system=5g:NoSchedule | | nautilus.io/ceph=true | nautilus.io/system=storage:NoSchedule | | nautilus.io/ceph-external=true | nautilus.io/system=storage:NoSchedule | | nautilus.io/jump=true | nautilus.io/system=jump:NoSchedule | | nautilus.io/perfsonar=true | nautilus.io/system=perfsonar:NoSchedule | | nautilus.io/linstor-server=true| nautilus.io/system=storage:NoSchedule | | nautilus.io/bluefield2=true | nautilus.io/reservation=bluefield2:NoSchedule | | nautilus.io/csusb=true | nautilus.io/reservation=csusb:NoSchedule | | nautilus.io/csu-tide=true | nautilus.io/reservation=csu-tide:NoSchedule | | nautilus.io/fpga-tutorial=true | nautilus.io/reservation=fpga-tutorial:NoSchedule | | nautilus.io/genai-lab=true | nautilus.io/reservation=genai-lab:NoSchedule | | nautilus.io/mizzou=true | nautilus.io/reservation=mizzou:NoSchedule | | msu-cache=true | nautilus.io/reservation=msu-cache:NoSchedule | | nautilus.io/prism-center=true | nautilus.io/reservation=prism-center:NoSchedule | | nautilus.io/qaic=undefined | nautilus.io/reservation=qaic:NoSchedule | | nautilus.io/reservation=cogrob | nautilus.io/reservation=cogrob:NoSchedule | | nautilus.io/reservation=wifire | nautilus.io/reservation=wifire:NoSchedule | | nautilus.io/nrp-llm=true | nautilus.io/reservation=nrp-llm:NoSchedule | | nautilus.io/sdsc-llm=true | nautilus.io/reservation=sdsc-llm:NoSchedule | | nautilus.io/sense=true | nautilus.io/reservation=sense:NoSchedule | | nautilus.io/stashcache=true | nautilus.io/reservation=osdf:NoSchedule | | nautilus.io/suncave=true | nautilus.io/reservation=suncave:NoSchedule | | nautilus.io/suncave-head=true | nautilus.io/reservation=suncave-head:NoSchedule | | um-cache=true | nautilus.io/reservation=um-cache:NoSchedule | | nautilus.io/arm64=true | nautilus.io/hardware=arm64:NoSchedule | | nautilus.io/large-gpu=true | nautilus.io/hardware=large-gpu:NoSchedule | | nautilus.io/disk-swap=true | nautilus.io/issue=disk-swap:NoSchedule | | nautilus.io/gitlab-issue=1234 | nautilus.io/issue=1234:NoSchedule | | nautilus.io/slow-network=true | nautilus.io/issue=slow-network:NoSchedule | | nautilus.io/testing=true | nautilus.io/issue=testing:NoSchedule | | nautilus.io/upgrading=true | nautilus.io/issue=upgrading:NoSchedule | | node.kubernetes.io/unreachable=undefined | node.kubernetes.io/unreachable=undefined:NoSchedule | [Observable notebook with taints summary](https://observablehq.com/d/1cb451398e09c3ff) ## Reservations **Groups may request exclusive access to entire nodes** if their workloads justify it. Such nodes can be reserved by setting the following `taint` and corresponding `toleration`: ```yaml spec: tolerations: - key: "nautilus.io/reservation" operator: "Equal" value: "group1" effect: "NoSchedule" ``` Please fill out the [node reservation form](https://nrp.ai/reservations/) if your group has a use case that would benefit from whole-node reservations. In addition, our cluster contains several sets of nodes dedicated to certain groups. Users can target **ONLY THE GROUP NODES** by using `affinity`, for example: ```yaml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: nautilus.io/reservation operator: In values: - group1 ``` For large jobs, this helps avoid consuming all shared cluster resources. Optionally, a higher priority can be used (contact the admins before using one). ## Other taints Some nodes in the cluster don't have access to public Internet, and can only access educational network. They still can pull images from Docker Hub using a proxy. If your workload is not using the public Internet resources, you might tolerate the `nautilus.io/science-dmz` and get access to additional nodes. --- # Faster Image Download - URL: https://nrp.ai/documentation/userdocs/running/fast-img-download/ - Section: User Guide > Running > Advanced patterns - Description: Speed up pod starts with spegel peer image caching, and notes on the deprecated Uber Kraken setup. ## Spegel image cache We're currently using [spegel](https://github.com/XenitAB/spegel) on most nodes running containerd. It currently does not support CRI-O (several nodes in the cluster). Spegel allows all nodes act as a caching proxy for images, which speeds up images downloads a lot. **UBER Kraken is now deprecated! If you used "localhost:30081" URLs, please switch back to normal ones.** --- # Globus Connect - URL: https://nrp.ai/documentation/userdocs/running/globus-connect/ - Section: User Guide > Running > Advanced patterns - Description: Stand up a personal Globus Connect endpoint with the ready-made recipe from the platform GitLab. We have a [globus-connect][1] Gitlab repository that provides step by step instructions and all needed files for running globus-connect container and creating a personal globus endpoint. [1]: https://gitlab.nrp-nautilus.io/prp/globus-connect --- # Python k8s API - URL: https://nrp.ai/documentation/userdocs/running/kubernetes/ - Section: User Guide > Running > Advanced patterns - Description: Drive Nautilus programmatically from inside the cluster with the Python Kubernetes client. Kubernetes offers API clients that allow you to build integrations under-the-hood of applications. The following is an example Python client for Nautilus that can submit it's own jobs and all the other functions of `kubectl` The config mounts the Ceph Shared FileSystem as well as the node's local scratch space. Be sure to configure resources appropriately to your application. > **Ephemeral storage on Nautilus:** Pods that write more than **50Gi** of ephemeral scratch data per container (for example, to an `emptyDir` at `/mnt/data`) can be evicted. When your code uses `emptyDir` or other scratch space, set `resources.requests.ephemeral-storage` in the container (and optionally `resources.limits.ephemeral-storage`) to the largest scratch size you expect (as in the example below). See [Local Scratch](https://nrp.ai/documentation/userdocs/storage/local) for more details and complete YAML. ``` import os from kubernetes import client, config, utils from kubernetes.client.rest import ApiException from skimage.transform import resize from uuid import uuid4 import time from pathlib import Path def test(): try: config.load_kube_config() v1 = client.CoreV1Api() ret = v1.list_namespaced_pod('ncmir-mm') except: ValueError('Could not connect to kube cluster') class Constants(object): NAMESPACE = YOUR_NAMESPACE class KubernetesApiClient(object): def __init__(self): # load print("\n Loading Nautilus Client... \n") def create_batch_api_client(self): return client.BatchV1Api(client.ApiClient()) def create_job_object(self, job_name, container_image, args=[],cmd = ['/bin/bash'], min_cpu=1, min_ram = 4, max_cpu=2, max_ram=12): res = client.V1ResourceRequirements( requests={"cpu":"1","memory":"8Gi","ephemeral-storage": "4Gi"}, limits = {"cpu":"4","memory":"24Gi","ephemeral-storage": "16Gi"}) volume_mount_2 = client.V1VolumeMount( mount_path='/ceph', name='ceph' ) volume_mount_1 = client.V1VolumeMount( mount_path='/mnt/data', name='data' ) #env = client.V1EnvVar(name='GOOGLE_APPLICATION_CREDENTIALS',value=google_app_credentials_path) container = client.V1Container( name=job_name, command = cmd, image=container_image, args=args, volume_mounts=[volume_mount_1,volume_mount_2], env=[], image_pull_policy="Always", resources = res) volume_1 = client.V1Volume( name='data' ) flex_2 = client.V1FlexVolumeSource( driver='ceph.rook.io/rook', fs_type='ceph', options = {'fsName': 'nautilusfs', 'clusterNamespace': 'rook', 'path': 'YOUR_CEPHFS_MOUNT', 'mountUser': 'YOUR_NAMESPACE', 'mountSecret': 'YOUR_CEPHFS_SECRET'} ) volume_2=client.V1Volume( name = 'ceph', flex_volume=flex_2 ) template = client.V1PodTemplateSpec( metadata=client.V1ObjectMeta(labels={"app": "sample"}), spec=client.V1PodSpec(restart_policy="Never", containers=[container], volumes=[volume_1,volume_2]) ) spec = client.V1JobSpec( template=template, backoff_limit=6, ttl_seconds_after_finished=60) job = client.V1Job( api_version="batch/v1", kind="Job", metadata=client.V1ObjectMeta(name=job_name), spec=spec) return job def submit_job(jobname, image = 'YOUR_IMAGE',args = []): api_client = KubernetesApiClient() job_api_client = api_client.create_batch_api_client() job = api_client.create_job_object(jobname, image, args) try: api_response = job_api_client.create_namespaced_job( namespace=Constants.NAMESPACE, body=job) print(str(api_response.status)) except ApiException as e: print(e) # Handle the exception. return job_api_client ``` --- # Federation - URL: https://nrp.ai/documentation/userdocs/running/federation/ - Section: User Guide > Running > Advanced patterns - Description: Extend a namespace to your own cluster with Kubernetes federation so jobs can run on both. ## Before Start Create a namespace on your cluster with the **same** name as the one on Nautilus. Otherwise, federation wont work. ```bash kubectl create namespace ``` (optional) set a new context with the new namespace on your cluster. ```bash kubectl config set-context --namespace= \ --cluster= \ --user= ``` ## On Nautilus Cluster (Target) First, change the currrent context to use your namespace inside the Nautilus cluster. ### Create a service account in Nautilus cluster for your cluster to access ```bash kubectl create serviceaccount my-sa ``` Nautilus cluster will automatically create a secret token associated with the service account you just created. ```bash kubectl get secret NAME TYPE DATA AGE default-token-j29lh kubernetes.io/service-account-token 3 2y178d kubernetes-dashboard-key-holder Opaque 2 42d my-sa-token-qfz8b kubernetes.io/service-account-token 3 11s ``` ### Create a config file named `config_sa` that includes the token emitted for the service account that you just created First get the name of the secret: ```bash TOKENNAME=`kubectl get serviceaccount/my-sa -o jsonpath='{.secrets[0].name}'` echo $TOKENNAME ``` Then get the secret using the `TOKENNAME` we found: ```bash TOKEN=`kubectl get secret $TOKENNAME -o jsonpath='{.data.token}'| base64 --decode` echo $TOKEN ``` After that copy the OIDC config file you use to access Nautilus cluster and add the new user instead of the current one. Change `` and `` accordingly. ```bash cd ~ cp .kube/ .kube/config_sa kubectl --kubeconfig=.kube/config_sa config set-credentials my-sa --token=$TOKEN kubectl --kubeconfig=.kube/config_sa config set-context --current --user=my-sa kubectl --kubeconfig=.kube/config_sa config view kubectl --kubeconfig=.kube/config_sa config unset users.http://cilogon.org/server ``` Now you need to let the service account act on behalf of user. To do this, change `` accordingly and run: ```bash kubectl create rolebinding my-sa-rb --clusterrole=edit --serviceaccount=:my-sa ``` Now check if you can list pods: ```bash kubectl --kubeconfig=.kube/config_sa get pods ``` The resulting `config_sa` file looks like: ```yaml apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: name: nautilus contexts: - context: cluster: nautilus namespace: user: my-sa name: nautilus current-context: nautilus kind: Config preferences: {} users: - name: my-sa user: token: REDACTED ``` ### Create a `Source` object in the same namespace. Use the same service account name you used in the previous step ```yaml #source.yaml apiVersion: multicluster.admiralty.io/v1alpha1 kind: Source metadata: name: my-cluster #name of this source object spec: serviceAccountName: my-sa ``` ```bash kubectl apply -f source.yaml ``` ## On Your Cluster (Source) First, change the currrent context to use your namespace (mine is `default`) inside your cluster. ### Create the secret holding credentials to access target cluster Encode the config file you just created in Base64 format and copy the output of the encoded config file. ```bash cat ~/.kube/config_sa | base64 -w 0 ``` Then create a `secret` object and paste the encoded config file into the `config` field. ```yaml #secret.yaml apiVersion: v1 data: config: kind: Secret metadata: name: my-secret type: Opaque ``` ```bash kubectl apply -f secret.yaml ``` ### Create a `Target` object, referencing the secret we just created ```yaml #target.yaml apiVersion: multicluster.admiralty.io/v1alpha1 kind: Target metadata: name: nautilus-cluster #name of this target object spec: kubeconfigSecret: name: my-secret ``` ```bash kubectl apply -f target.yaml ``` ### Label the namespace as being federated `kubectl label ns default multicluster-scheduler=enabled` ### Check if the virual node is up ```bash kubectl get nodes --watch NAME STATUS ROLES AGE VERSION admiralty-dev-namespace-nautilus-tg-2e2a858480 Ready cluster,control-plane,master 4d23h ``` ### Try to run a federated pod by adding the annotation ```yaml #test-pod.yaml apiVersion: v1 kind: Pod metadata: annotations: multicluster.admiralty.io/elect: '' name: test-pod spec: containers: - name: mypod image: centos:centos7 resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: ['sh', '-c', "echo 'Im a new pod' && sleep infinity"] ``` ```bash kubectl apply -f test-pod.yaml ``` ### Check if the proxy and delegate pods are running on source and target cluster respectively ```bash #proxy pod kubectl --context= get pods NAME READY STATUS RESTARTS AGE test-pod 1/1 Running 0 20m ``` ```bash #delegate pod kubectl --context=nautilus get pods NAME READY STATUS RESTARTS AGE test-pod-stdhg 1/1 Running 0 15m ``` --- # GUI Desktop - URL: https://nrp.ai/documentation/userdocs/running/gui-desktop/ - Section: User Guide > Running > Advanced patterns - Description: Deploy a browser-based GUI desktop pod in your own namespace; Coder and JupyterHub are the preferred routes. **Note that Coder () and JupyterHub West () are currently the preferred methods to deploy the GUI Desktop containers. Use the below instructions if you want to deploy in your own namespace.** The default login for Selkies in the browser is the username `ubuntu` with the password `mypasswd`, or whatever `PASSWD` is set to. With [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop) or [docker-selkies-egl-desktop](https://github.com/selkies-project/docker-selkies-egl-desktop), users may start a KDE Plasma desktop accelerated with NVIDIA GPUs, streamed from the NRP to your web browser by [Selkies](https://github.com/selkies-project/selkies). Both containers support OpenGL and Vulkan, and both come with Firefox, Google Chrome, LibreOffice, VLC, Steam, Wine, and an apps panel in the Selkies dashboard that installs portable applications into the home directory. [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop) runs an X.Org server of its own on the GPU with the NVIDIA driver, so OpenGL and Vulkan reach applications through the vendor's own driver with nothing translating in between. It provides the best performance and is generally recommended when the pod has a GPU to itself, which a `nvidia.com/gpu` request gives it. The first start takes a minute longer while the container fetches the X server modules from the NVIDIA driver installer matching the node's driver. A GPU without a display device, such as the Tesla P100, is served on the Selkies base container's framebuffer server instead, which the container log says. [docker-selkies-egl-desktop](https://github.com/selkies-project/docker-selkies-egl-desktop) reaches the GPU through EGL without an X.Org server, on the display servers of the Selkies base container. It is versatile in various environments: it also runs without a GPU (in software, with the x264 encoder), sharing a GPU with multiple containers is possible, the desktop runs on a Wayland session with `SELKIES_WAYLAND=true`, and it is also possible to be used in HPC clusters with [Apptainer](https://apptainer.org) as described in the [Selkies documentation](https://docs.selkies.io/latest/start/#apptainer). Both images are built on Ubuntu 26.04 and published with the tags `26.04` and `latest` for the current build, and persistent tags in the format `26.04-20260101010101` for a specific build. **Support is available in [#ue4research:matrix.nrp-nautilus.io](https://matrix-to.nrp-nautilus.io/#/#ue4research:matrix.nrp-nautilus.io) (use [this link](https://matrix.to/#/#ue4research:matrix.nrp-nautilus.io) if you use a matrix.org account), just ask your questions there.** Upstream support for the containers is on the Selkies [Discord](https://discord.gg/wDNGDeSW5F) and the GitHub Discussions of each repository. Please give the repositories a star! ## Streaming Selkies streams over WebSockets by default, which needs nothing but the web port that the Ingress below exposes. The WebRTC transport is selected with `SELKIES_MODE` or switched to from the web interface, and uses the NRP TURN server configured in the reference configurations below. The video encoder, video bitrate, frame rate, audio bitrate, and UI scaling are chosen from the web interface, so the reference configurations leave them alone. The default encoder `h264enc` encodes on the GPU with NVENC when the pod has a GPU and with x264 on the CPU otherwise, so nothing has to be set for hardware encoding; `h265enc`, `vp8enc`, `vp9enc`, and `av1enc` run on the GPU where it carries the codec in the same way, `h264enc-striped` and `jpeg` are CPU encoders, and a browser that cannot decode the chosen codec steps through the ones it can. Do not set `SELKIES_ENCODER`: a single value locks the encoder to that codec and takes the choice away from the web interface. ### TURN Server The WebRTC transport relays through the NRP TURN server, `turn.nrp-nautilus.io`. Since the right TURN server closest to you leads to the lowest latency, use the command `ping turn.nrp-nautilus.io` on your client (install `iputils-ping` when using Linux if the `ping` command does not work) to check that your DNS is correctly configured. If the latency is considerably high (over 60-70 milliseconds) or the TURN server host node location is suspiciously far from both you and the Kubernetes node, consider changing your client DNS server to `8.8.8.8` and `8.8.4.4` or the DNS over HTTPS / DNS over TLS options [provided by Google](https://developers.google.com/speed/public-dns/docs/using). [CloudFlare's DNS](https://1.1.1.1/dns/) server (addresses `1.1.1.1` and `1.0.0.1`) may show issues locating the nearest relay server, which is crucial for performance. The reference configurations below take their TURN credentials from the TURN REST API credential endpoint `http://turn-rest.nrp-nautilus.io` (accessible only to Nautilus pods, username and password are valid for 24 hours, renewed every time `http://turn-rest.nrp-nautilus.io` is probed). If you are using the Nautilus coTURN server for other WebRTC workloads, use the same endpoint, and also add the below configuration (change DNS server as adequate) in your Kubernetes `.yml` configuration to minimize TURN server latency: ```yaml dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 - 8.8.4.4 ``` ## Usage The below is a reference configuration `xgl.yml` for [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop): ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: xgl spec: replicas: 1 selector: matchLabels: app: xgl template: metadata: labels: app: xgl spec: hostname: xgl containers: - name: xgl # Tags: `latest` or `26.04` for the current Ubuntu 26.04 build, and a persistent build such as `26.04-20260101010101` image: ghcr.io/selkies-project/selkies-glx-desktop:latest env: - name: TZ value: 'UTC' # Password of the container Linux user, and of the web login when `SELKIES_BASIC_AUTH_PASSWORD` is not set; choose either `value:` or `secretKeyRef:` but not both at the same time - name: PASSWD # value: 'mypasswd' valueFrom: secretKeyRef: name: my-pass key: my-pass # The X server's initial mode, replaced by the client's size once it connects (dynamic resizing is on by default); the size a manual resolution locks the stream to and the frame rate the stream starts at take its place where set - name: DISPLAY_SIZEW value: '1920' - name: DISPLAY_SIZEH value: '1080' - name: DISPLAY_REFRESH value: '60' # - name: SELKIES_MANUAL_WIDTH # value: '1920' # - name: SELKIES_MANUAL_HEIGHT # value: '1080' # - name: SELKIES_FRAMERATE # value: '60' # The video port the NVIDIA driver reports a monitor on: keep `DFP` on datacenter GPUs, use an empty `DP-*` port on consumer and professional GPUs for resolutions above 2560 x 1600 - name: VIDEO_PORT value: 'DFP' ### # Selkies settings, see `selkies --help` and https://github.com/selkies-project/selkies/blob/main/docs/settings.md for the rest ### # Transport: WebSockets by default, `webrtc` uses the NRP TURN server configured below; both can be switched from the web interface # - name: SELKIES_MODE # value: 'webrtc' # The web login, on by default: `ubuntu` and `PASSWD` unless set here - name: SELKIES_ENABLE_BASIC_AUTH value: 'true' # - name: SELKIES_BASIC_AUTH_USER # value: 'ubuntu' # - name: SELKIES_BASIC_AUTH_PASSWORD # valueFrom: # secretKeyRef: # name: my-pass # key: my-pass # The Ingress terminates TLS, so the container serves plain HTTP; `true` serves HTTPS on a certificate the container mints per install, or on one named with `SELKIES_HTTPS_CERT` and `SELKIES_HTTPS_KEY` - name: SELKIES_ENABLE_HTTPS value: 'false' # NRP TURN REST API for the WebRTC transport, which replaces every other TURN setting - name: SELKIES_TURN_REST_URI value: 'http://turn-rest.nrp-nautilus.io' # Change to `tcp` if the UDP protocol is throttled or blocked in your client network - name: SELKIES_TURN_PROTOCOL value: 'udp' # TURN over TLS, which the NRP TURN server supports, if your network inspects the WebRTC or STUN/TURN protocols - name: SELKIES_TURN_TLS value: 'false' stdin: true tty: true ports: - name: http containerPort: 8080 protocol: TCP resources: limits: memory: 64Gi cpu: '16' nvidia.com/gpu: 1 requests: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 volumeMounts: # The browsers inside the desktop crash on the default shared memory - mountPath: /dev/shm name: dshm - mountPath: /cache name: xgl-cache-vol - mountPath: /home/ubuntu name: xgl-root-vol # Steers the TURN server's location-aware DNS for the WebRTC transport; remove to keep the cluster's DNS dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 - 8.8.4.4 volumes: - name: dshm emptyDir: medium: Memory - name: xgl-cache-vol emptyDir: {} # persistentVolumeClaim: # claimName: xgl-cache-vol - name: xgl-root-vol emptyDir: {} # persistentVolumeClaim: # claimName: xgl-root-vol affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: NotIn values: - ucsd-suncave # - key: topology.kubernetes.io/region # operator: In # values: # - us-west ``` The below is a reference configuration `egl.yml` for [docker-selkies-egl-desktop](https://github.com/selkies-project/docker-selkies-egl-desktop): ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: egl spec: replicas: 1 selector: matchLabels: app: egl template: metadata: labels: app: egl spec: hostname: egl containers: - name: egl # Tags: `latest` or `26.04` for the current Ubuntu 26.04 build, and a persistent build such as `26.04-20260101010101` image: ghcr.io/selkies-project/selkies-egl-desktop:latest env: - name: TZ value: 'UTC' # Password of the container Linux user, and of the web login when `SELKIES_BASIC_AUTH_PASSWORD` is not set; choose either `value:` or `secretKeyRef:` but not both at the same time - name: PASSWD # value: 'mypasswd' valueFrom: secretKeyRef: name: my-pass key: my-pass ### # Selkies settings, see `selkies --help` and https://github.com/selkies-project/selkies/blob/main/docs/settings.md for the rest ### # Transport: WebSockets by default, `webrtc` uses the NRP TURN server configured below; both can be switched from the web interface # - name: SELKIES_MODE # value: 'webrtc' # `true` runs the desktop on the headless Wayland backend (nested kwin) instead of the X11 framebuffer server - name: SELKIES_WAYLAND value: 'false' # The size the X11 desktop has until a client connects and replaces it with its own (dynamic resizing is on by default); the size a manual resolution locks the stream to takes its place where set # - name: DISPLAY_SIZEW # value: '1920' # - name: DISPLAY_SIZEH # value: '1080' # - name: SELKIES_MANUAL_WIDTH # value: '1920' # - name: SELKIES_MANUAL_HEIGHT # value: '1080' # The web login, on by default: `ubuntu` and `PASSWD` unless set here - name: SELKIES_ENABLE_BASIC_AUTH value: 'true' # - name: SELKIES_BASIC_AUTH_USER # value: 'ubuntu' # - name: SELKIES_BASIC_AUTH_PASSWORD # valueFrom: # secretKeyRef: # name: my-pass # key: my-pass # The Ingress terminates TLS, so the container serves plain HTTP; `true` serves HTTPS on a certificate the container mints per install, or on one named with `SELKIES_HTTPS_CERT` and `SELKIES_HTTPS_KEY` - name: SELKIES_ENABLE_HTTPS value: 'false' # NRP TURN REST API for the WebRTC transport, which replaces every other TURN setting - name: SELKIES_TURN_REST_URI value: 'http://turn-rest.nrp-nautilus.io' # Change to `tcp` if the UDP protocol is throttled or blocked in your client network - name: SELKIES_TURN_PROTOCOL value: 'udp' # TURN over TLS, which the NRP TURN server supports, if your network inspects the WebRTC or STUN/TURN protocols - name: SELKIES_TURN_TLS value: 'false' stdin: true tty: true ports: - name: http containerPort: 8080 protocol: TCP resources: limits: memory: 64Gi cpu: '16' # Remove the two `nvidia.com/gpu` lines to run the desktop without a GPU nvidia.com/gpu: 1 requests: memory: 100Mi cpu: 100m nvidia.com/gpu: 1 volumeMounts: # The browsers inside the desktop crash on the default shared memory - mountPath: /dev/shm name: dshm - mountPath: /cache name: egl-cache-vol - mountPath: /home/ubuntu name: egl-root-vol # Steers the TURN server's location-aware DNS for the WebRTC transport; remove to keep the cluster's DNS dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 - 8.8.4.4 volumes: - name: dshm emptyDir: medium: Memory - name: egl-cache-vol emptyDir: {} # persistentVolumeClaim: # claimName: egl-cache-vol - name: egl-root-vol emptyDir: {} # persistentVolumeClaim: # claimName: egl-root-vol # affinity: # nodeAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # nodeSelectorTerms: # - matchExpressions: # - key: topology.kubernetes.io/region # operator: In # values: # - us-west ``` ### Customization In one entry, `value:` and `valueFrom:` must not exist at the same time. **Comment out `emptyDir: {}` and uncomment `persistentVolumeClaim:`, then change `claimName:` to the name of your PersistentVolumeClaim after uncommenting.** `rook-ceph-block-[region]` or `linstor-[region]` are the recommended StorageClasses to be mounted to `/home/ubuntu`. However, if their performances are slow for sequential read/writes, you may use `rook-cephfs-[region]` but **NEVER MOUNT TO `/home/ubuntu`**. Instead mount to a different directory such as `/home/ubuntu/persistent` or `/mnt/persistent`. Refer to the [Storage](https://nrp.ai/documentation/userdocs/storage/intro/) section for more information. To run on a specific type of GPU, add the node affinity described in [GPU pods](https://nrp.ai/documentation/userdocs/running/gpu-pods/) to the reference configuration. `DISPLAY_SIZEW` and `DISPLAY_SIZEH` set the size the desktop has until your browser connects, and dynamic resizing then gives it the size of your browser window; `DISPLAY_REFRESH` sets the refresh rate the X server of [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop) starts at. Where a Selkies setting names the same thing, the Selkies setting takes their place, read exactly as Selkies reads it: `SELKIES_MANUAL_WIDTH` and `SELKIES_MANUAL_HEIGHT` lock the desktop to a fixed size instead of following your browser window, and `SELKIES_FRAMERATE` sets the frame rate the stream starts at, which that X server then starts at too. There is no color depth setting: the desktop is always 24-bit, the depth Selkies captures. The two settings below apply to the WebRTC transport: **If your client network blocks (likely your campus network) or throttles (likely your home network restricted by your Internet Service Provider) the UDP protocol, change the environment variable `SELKIES_TURN_PROTOCOL` to `tcp`.** **If your network or country enforces [Deep Packet Inspection](https://en.wikipedia.org/wiki/Deep_packet_inspection) to the WebRTC or STUN/TURN protocols, set `SELKIES_TURN_TLS` to `true`.** Check the `README.md` of both GitHub repositories [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop) or [docker-selkies-egl-desktop](https://github.com/selkies-project/docker-selkies-egl-desktop), as well as [Selkies](https://github.com/selkies-project/selkies) and its [settings reference](https://github.com/selkies-project/selkies/blob/main/docs/settings.md), for more details on configuration customization. Please give the repositories a star too! ### Secret Generation Replace `YOUR_PASSWORD` with the password for the container that you intend to use. Replace `my-name` and `my-key` with the name of the secret you want to use for your password. If you want to use a different name and key, make sure to update the above `xgl.yml` or `egl.yml` files as well as the below command. After customizing, run the below command in edited form: ```bash kubectl create secret generic my-name --from-literal=my-key=YOUR_PASSWORD ``` The above command should be used in conjunction to the below `xgl.yml`/`egl.yml` configuration (edit as adequate): ```yaml env: - name: PASSWD valueFrom: secretKeyRef: name: my-name key: my-key ``` ### Container Start After saving and editing the reference configurations, run either of these commands to start the container depending on the type of your container: ```bash kubectl create -f xgl.yml ``` ```bash kubectl create -f egl.yml ``` ## Exposing the Container The below reference configuration `xgl-ingress.yml` is to expose your [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop) container to the `*.nrp-nautilus.io` endpoint. Replace `YOUR_ENDPOINT` to the subdomain you want to use. The Ingress terminates TLS, which is why the reference configurations above set `SELKIES_ENABLE_HTTPS` to `false`. Modify the configuration as in [Scaling and exposing](https://nrp.ai/documentation/userdocs/tutorial/basic2) to customize when there are multiple desktop deployments in a namespace. You can just use `kubectl port-forward deployment/xgl 8080:8080` and access `http://localhost:8080`, but this will likely have higher latency and subpar performance. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: xgl spec: ingressClassName: haproxy rules: - host: YOUR_ENDPOINT.nrp-nautilus.io http: paths: - backend: service: name: xgl port: name: http path: / pathType: ImplementationSpecific tls: - hosts: - YOUR_ENDPOINT.nrp-nautilus.io --- apiVersion: v1 kind: Service metadata: name: xgl labels: app: xgl spec: selector: app: xgl ports: - name: http protocol: TCP port: 8080 ``` If you are deploying multiple instances in one namespace, you must change `backend:`, `selector:`, and `labels:` to the name of your `Deployment` and `Service`. Run the below command after saving the changed reference configuration file: ```bash kubectl create -f xgl-ingress.yml ``` Access `YOUR_ENDPOINT.nrp-nautilus.io` with your web browser. **The username is `ubuntu` and the password is the `my-name`/`my-key` secret that you have set.** The below reference configuration `egl-ingress.yml` is to expose your [docker-selkies-egl-desktop](https://github.com/selkies-project/docker-selkies-egl-desktop) container to the `*.nrp-nautilus.io` endpoint. Replace `YOUR_ENDPOINT` to the subdomain you want to use. Modify the configuration as in [Scaling and exposing](https://nrp.ai/documentation/userdocs/tutorial/basic2) to customize when there are multiple desktop deployments in a namespace. You can just use `kubectl port-forward deployment/egl 8080:8080` and access `http://localhost:8080`, but this will likely have higher latency and subpar performance. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: egl spec: ingressClassName: haproxy rules: - host: YOUR_ENDPOINT.nrp-nautilus.io http: paths: - backend: service: name: egl port: name: http path: / pathType: ImplementationSpecific tls: - hosts: - YOUR_ENDPOINT.nrp-nautilus.io --- apiVersion: v1 kind: Service metadata: name: egl labels: app: egl spec: selector: app: egl ports: - name: http protocol: TCP port: 8080 ``` If you are deploying multiple instances in one namespace, you must change `backend:`, `selector:`, and `labels:` to the name of your `Deployment` and `Service`. Run the below command after saving the changed reference configuration file: ```bash kubectl create -f egl-ingress.yml ``` Access `YOUR_ENDPOINT.nrp-nautilus.io` with your web browser. **The username is `ubuntu` and the password is the `my-name`/`my-key` secret that you have set.** ## Reproducibility in Containers Reproducibility is important, but the `latest` and `26.04` containers are consistently updated to fix various issues that arise and add improvements. Because some changes are radical and change the fundamental structure of the container (normally when there is a dead-end with breaking issues or breakable upstream changes), they might break for your specific workflow. Since you would not want to see your container break one day because there was a change, you could use persistent container tags (in the format `26.04-20260101010101`) with the time which a commit was made, if you want to base this container to build a customized container. Because the containers have rolling releases, container versions are tagged by the time of which a commit has been made. You may access such available persistent container tags in and . It is **STRONGLY** recommended that you update the container tags frequently or use `apt-get upgrade` in every new build where this container has been based (but this may also break the container in a few edge cases), as many serious security vulnerabilities are frequently present in old containers. Conversely, completely fresh builds from the `Dockerfile` with older commits may and will break at any time and thus the latest `Dockerfile` should be used. ## Container Customization You can import either `https://github.com/selkies-project/docker-selkies-glx-desktop.git` or `https://github.com/selkies-project/docker-selkies-egl-desktop.git` by importing with **Repo by URL** from . Refer to [Building in GitLab](https://nrp.ai/documentation/userdocs/development/gitlab) on how you can change and build your own customized container. The containers are built on Ubuntu 26.04 only; `--build-arg="BASE_IMAGE="` after `/kaniko/executor` in the example `.gitlab-ci.yml` file names the Ubuntu 26.04 Selkies base container they build on. **If you want to change the `Dockerfile`, you are recommended to use the original container as a base container and only replace the entrypoint scripts and the [s6](https://skarnet.org/software/s6/) service files under `/etc/service/`. This will keep you up to date with the latest updates. Use persistent container tags (such as `26.04-20260101010101`) to preserve a specific container build.** Start with the below sample `Dockerfile` example and place your modified `container-entrypoint.sh` and s6 service files within the same empty directory or Git repository (switch the `FROM` line to `ghcr.io/selkies-project/selkies-glx-desktop:${DISTRIB_RELEASE}` or `ghcr.io/selkies-project/selkies-egl-desktop:${DISTRIB_RELEASE}`). The containers are rootless, so a layer that installs packages releases the setuid files that `dpkg` replaces, installs under `fakeroot`, and restores them, as in the commented lines: ```dockerfile ARG DISTRIB_RELEASE=26.04 FROM ghcr.io/selkies-project/selkies-glx-desktop:${DISTRIB_RELEASE} ARG DISTRIB_RELEASE USER 0 SHELL ["/bin/sh", "-c"] # Replace changed files # Copy scripts and service definitions used to start the container with `--chown=1000:1000` #COPY --chown=1000:1000 container-entrypoint.sh /etc/container-entrypoint.sh #RUN chmod -f 755 /etc/container-entrypoint.sh #COPY --chown=1000:1000 selkies-entrypoint.sh /etc/selkies-entrypoint.sh #RUN chmod -f 755 /etc/selkies-entrypoint.sh # Replace or add s6 services (one directory per service under /etc/service) #COPY --chown=1000:1000 services/ /etc/service/ #RUN find /etc/service -name run -exec chmod -f 755 {} + # Install packages #RUN selkies-privileged-files release #USER 1000 #SHELL ["/usr/bin/fakeroot", "--", "/bin/sh", "-c"] #RUN apt-get update && apt-get install --no-install-recommends -y #USER 0 #SHELL ["/bin/sh", "-c"] #RUN selkies-privileged-files restore USER 1000 ENV SHELL=/bin/bash ENV USER=ubuntu ENV HOME=/home/ubuntu WORKDIR /home/ubuntu EXPOSE 8080 ENTRYPOINT ["/etc/container-entrypoint.sh"] ``` ## Conclusion Again, please give [docker-selkies-glx-desktop](https://github.com/selkies-project/docker-selkies-glx-desktop), [docker-selkies-egl-desktop](https://github.com/selkies-project/docker-selkies-egl-desktop), and [Selkies](https://github.com/selkies-project/selkies) a star if the containers were useful to you. --- # Scientific Images - URL: https://nrp.ai/documentation/userdocs/running/sci-img/ - Section: User Guide > Running > Advanced patterns - Description: Pick a ready scientific Python, R, or notebook image for a pod or JupyterHub from the stacks we mirror. This page is mostly related to our [Official JupyterHub](https://jupyterhub-west.nrp-nautilus.io), but all images can be also used in other pods deployed directly on the cluster. There are two main projects providing the stack of images: [Docker Stack](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html) [B-Data](https://gitlab.b-data.ch/jupyterlab) Both projects now support CUDA and ARM. Docker stack only has CUDA in TensorFlow and PyTorch variants. **The list of NRP-provided images and registry links are available in our [GitLab registry](https://gitlab.nrp-nautilus.io/nrp/scientific-images)**: The **NRP image with additional libraries** (gitlab-registry.nrp-nautilus.io/nrp/scientific-images/python) is based on Docker Stack TensorFlow and PyTorch, with additional packages. NOTE: If you are using the VS Code editor with a Jupyter Notebook, use the `base` Python Environment as the kernel. Moreover, do not use the `/usr/bin/python`/`/usr/bin/python3` Python executable. Instead, use `python` or `python3` without the path. Refer to for the list of packages installed. We can add more libraries to this image by requesting in [Nautilus Support](https://nrp.ai/contact). The Desktop image has the X11 Window system installed and you can launch the GUI interface in Jupyter with this image. It's based on the Minimal stack. The **Selkies Desktop** image (gitlab-registry.nrp-nautilus.io/nrp/scientific-images/desktop/selkies) streams a KDE Plasma desktop from your server with [Selkies](https://github.com/selkies-project/selkies): the **Selkies** item in the JupyterLab launcher opens it in a new tab, signed in through JupyterHub with no password of its own, and it encodes on the GPU when your server has one. --- # Exposing HTTP - URL: https://nrp.ai/documentation/userdocs/running/ingress/ - Section: User Guide > Running > Exposing apps - Description: Expose a pod HTTP service at a public nrp-nautilus.io address with an Ingress. While pods are not accessible from outside the cluster, you can expose the http services provided by pods by using the [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) controllers. In general we don't allow exposing non-http applications via TCP ports, but if you really need to do that, contact us on [Nautilus Support](https://nrp.ai/contact). For a complete example see the ([Tutorial](https://nrp.ai/documentation/userdocs/tutorial/basic2/)) Refer to the [HAProxy Ingress Documentation](https://haproxy-ingress.github.io/docs/configuration/keys/) to set up additional config. **The annotation to use is either `haproxy-ingress.github.io` or `ingress.kubernetes.io`. DO NOT use the `haproxy.org` annotation to configure the Ingress.** To expose a port in your pod, you first need to create a service for it. For the pod ```yaml apiVersion: v1 kind: Pod metadata: name: my-pod labels: k8s-app: test-http spec: containers: - name: mypod image: nginxdemos/hello:plain-text ``` The service might look like: ```yaml apiVersion: v1 kind: Service metadata: labels: k8s-app: test-svc name: test-svc spec: ports: - port: 8080 protocol: TCP targetPort: 80 selector: k8s-app: test-http type: ClusterIP ``` Where `spec.selector.