Fixing ACCESS-OM3 'Illegal Instruction' Errors On Setonix

Welcome, fellow researchers and HPC enthusiasts! If you've landed here, chances are you're wrestling with the cryptic 'Illegal instruction (core dumped)' error while trying to run your ACCESS-OM3 simulations on Setonix Pawsey. This guide is your ultimate weapon against this common but perplexing supercomputing challenge. We're here to demystify this error and provide you with clear, actionable steps to get your climate models back on track and computing efficiently on one of Australia's most powerful HPC systems. Understanding the root cause is the first step, and we'll delve deep into CPU architecture mismatches and compiler flags that often lead to this headache. Whether you're running the dev-MC_100km_jra_ryf configuration or another complex model, the principles for resolving this issue remain the same. Let's conquer this technical hurdle together and ensure your valuable research time isn't lost to avoidable technical glitches.

What's Up with 'Illegal Instruction' on Setonix, Guys?

Alright, listen up, folks! You've hit that dreaded 'Illegal instruction (core dumped)' error when trying to run your ACCESS-OM3 simulation, specifically the dev-MC_100km_jra_ryf configuration, on the mighty Setonix Pawsey supercomputer. Trust me, you're not alone, and it's one of those head-scratching moments that can throw a wrench into your research flow. But don't sweat it too much; we're going to break down exactly what this error means and, more importantly, how to squash it like a bug. At its core, an illegal instruction error means your program, in this case, the access-om3-MOM6-CICE6 executable, tried to tell the CPU to do something it simply doesn't understand or isn't capable of performing. Think of it like trying to speak French to someone who only understands Mandarin – total communication breakdown! This isn't just a random glitch; it's a fundamental incompatibility between the compiled code and the underlying hardware. When we talk about high-performance computing (HPC) environments like Setonix, we're dealing with cutting-edge processors that have specific instruction sets. These are like specialized toolkits for the CPU, designed to make calculations super-fast. If your ACCESS-OM3 executable was built using a compiler that assumed a different, perhaps older or newer, set of tools than what the Setonix compute nodes actually possess, then BAM! You get an illegal instruction. The additional '(core dumped)' part is the CPU's way of saying, "I crashed, and I've left a 'core' file behind with all my memory contents at the time of the crash." This file, while intimidating, is actually a treasure trove for deep debugging, though often the illegal instruction itself gives us enough clues to solve the puzzle without needing to delve into the core dump's raw data. For researchers working with complex climate models like ACCESS-OM3, which simulate Earth's climate system using components like MOM6 (ocean model) and CICE6 (sea ice model), these errors can be particularly frustrating because they often appear after a long, successful compilation process. The dev-MC_100km_jra_ryf configuration indicates a specific development branch for a 100km resolution model using JRA-55 atmospheric forcing and an eddy-rich ocean. This level of complexity means that any underlying architectural mismatch can propagate quickly, causing the entire simulation to halt. Understanding this fundamental issue is the first crucial step in getting your ACCESS-OM3 jobs running smoothly on Setonix. We'll guide you through checking your setup and identifying where this instruction set miscommunication might be happening, turning this frustration into a learning opportunity.

Diving Deep into the Setup: Your Setonix Slurm Script

Alright, let's get our hands dirty and dissect your runom3.sh script, because sometimes the devil is in the details, even if the script itself isn't the direct cause of the 'illegal instruction' error. This script is your instruction manual for Setonix's Slurm workload manager, telling it how to allocate resources and what to run. You've clearly followed the Pawsey guidelines for Slurm Batch Scripts on Setonix, which is a solid starting point for any HPC job submission. Let's break down what you've got here, and why it generally looks good for an MPI job, while also highlighting points that are crucial for context.

First up, the #!/bin/bash --login line is standard practice, ensuring your shell environment is properly set up with all necessary paths and variables upon job start. Then come the all-important SLURM directives, prefixed with #SBATCH. These are your requests to the scheduler, laying out the resources needed for your ACCESS-OM3 simulation:

• #SBATCH --account=pawsey0889: This correctly specifies your project account. It's an essential directive for resource allocation on Setonix; without it, your job wouldn't even be queued. Always ensure this matches your active Pawsey project.
• #SBATCH --partition=work: You're submitting to the work partition, which is typically designated for general compute tasks, development, and shorter runs. This is a common and appropriate choice for testing or debugging before moving to larger, longer runs in other partitions.
• #SBATCH --ntasks=512: Here you're requesting 512 MPI tasks. This is a significant number, indicating a large-scale simulation, typical for complex ACCESS-OM3 models that require substantial parallelism. For MPI-intensive codes, balancing tasks across nodes is critical.
• #SBATCH --ntasks-per-node=128: This directive tells Slurm to place 128 tasks on each allocated node. Given that Setonix nodes typically feature 128 physical cores (on its AMD EPYC 'Milan' generation nodes, or 128/256 for 'Genoa' depending on core/thread configuration), this suggests you're aiming to utilize all available cores on each node – a common and efficient strategy for maximizing throughput in MPI-based applications like ACCESS-OM3. This means your job will be allocated 512 / 128 = 4 compute nodes.
• #SBATCH --exclusive: Requesting exclusive access to the nodes is a smart move for MPI jobs that demand full network bandwidth and minimal interference. This prevents other users' jobs from sharing your nodes, thereby minimizing contention for CPU, memory, and network resources, which can significantly improve performance consistency and reduce variability in your ACCESS-OM3 runs.
• #SBATCH --time=00:01:00: This is where things get interesting for debugging. You've set a very short wall-clock time limit of just 1 minute. While too short for a full ACCESS-OM3 run, this minimal time limit is actually invaluable for debugging. It allows you to quickly test if a job starts or immediately crashes without consuming significant allocation hours on failing jobs. For production runs, you'd extend this considerably.

Next, you've included some critical MPI related environment variables. These are often crucial for optimizing communication and debugging MPI applications on HPC interconnects:

• export MPICH_OFI_STARTUP_CONNECT=1 and export MPICH_OFI_VERBOSE=1: These variables relate to the OpenFabrics Interface (OFI), which is a high-performance network interface commonly used in HPC to allow applications to directly access network hardware with low latency and high bandwidth. Setting STARTUP_CONNECT to 1 can sometimes aid in establishing connections reliably across multiple nodes, especially in complex network topologies. VERBOSE=1 is a fantastic debugging tool as it forces MPICH to print out a lot more information about its internal workings during startup and execution. This verbosity can sometimes reveal subtle issues with network setup, fabric initialisation, or resource allocation that might otherwise remain hidden.
• export MPICH_ENV_DISPLAY=1 and export MPICH_MEMORY_REPORT=1: These are also excellent for debugging. MPICH_ENV_DISPLAY=1 forces MPICH to print all its environment variables at startup, giving you a clear snapshot of the environment MPI is operating within. This helps verify that all necessary paths and settings are correctly propagated. MPICH_MEMORY_REPORT=1 provides details on memory usage by MPI processes, which, while not directly related to an illegal instruction, can be useful for general MPI application tuning, identifying memory leaks, and ensuring stable operation on Setonix by preventing out-of-memory errors.

Finally, the Run the desired code section:

• srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6: This is the command that actually launches your ACCESS-OM3 executable. You're using srun, the standard Slurm command to run parallel jobs, and correctly passing the number of nodes ($SLURM_JOB_NUM_NODES, which will resolve to 4 in your case) and total tasks ($SLURM_NTASKS, which is 512). The specific path to your executable, /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6, is very telling. It clearly indicates that your ACCESS-OM3 application was built and managed using Spack, a powerful and increasingly popular package manager for HPC environments. This Spack environment is a key piece of the puzzle for understanding the illegal instruction error.

Overall, your runom3.sh script is well-structured, adheres to Setonix guidelines, and utilizes appropriate parameters for a large-scale MPI simulation. However, the script itself doesn't inherently cause an illegal instruction error; it merely tries to execute a program. This means the problem lies not in how you're running it, but what you're running – specifically, how access-om3-MOM6-CICE6 was compiled and for which target architecture. The Spack environment path strongly suggests that the build process, which happens before this script is ever run, is where we need to focus our attention, specifically regarding compiler flags and target CPU architecture compatibility on Setonix.

The Core Problem: CPU Architecture Mismatch and Compiler Flags

Okay, guys, if your Slurm script looks good but you're still hitting that dreaded 'Illegal instruction (core dumped)' error on Setonix, then it's time to zero in on the real culprit: a CPU architecture mismatch or, more specifically, issues with your compiler flags during the build process. This is, without a shadow of a doubt, the most common reason for this particular error in High-Performance Computing (HPC) environments like Setonix Pawsey. Let me explain why this happens and why it's so critical for ACCESS-OM3.

Modern CPUs, like the AMD EPYC processors powering Setonix, come with incredibly sophisticated instruction sets. These aren't just basic 'add' or 'multiply' operations; they include advanced vector instructions like AVX, AVX2, and AVX-512 (and even newer ones like VNNI and BFloat16, primarily for AI/ML workloads), designed to perform multiple operations simultaneously on large chunks of data. These advanced instructions are precisely what make scientific applications, especially computationally intensive climate models like ACCESS-OM3, run at blistering speeds. When you compile your ACCESS-OM3 code, the compiler (like GCC, Intel, or AOCC on Setonix) translates your human-readable C/C++/Fortran code into machine-specific instructions. The problem arises when the compiler optimizes this code for a specific set of CPU features or a particular CPU generation, and then you try to run that compiled executable on a CPU that doesn't possess those features. The moment the program encounters an instruction it doesn't recognize or can't execute, the CPU triggers an 'illegal instruction' fault, leading to the program's immediate termination and the generation of a 'core dump'.

Consider Setonix Pawsey: it's a powerful system predominantly built with AMD EPYC processors. Specifically, Setonix utilizes a mix of AMD EPYC Milan (Zen 3) and Genoa (Zen 4) architectures across its compute nodes. While both are highly capable processors, they have distinct instruction set capabilities and microarchitectural differences. If your ACCESS-OM3 executable, access-om3-MOM6-CICE6, was compiled assuming, say, a Zen 4 processor with certain advanced vector instructions enabled by default or explicitly through compiler flags (like AVX-512 for specific operations), but then it ends up running on a Zen 3 node that lacks those specific instructions, the CPU will encounter an 'illegal instruction' and promptly crash. This is exactly what the nid001879, nid001923, nid001925 errors in your slurm-34894388.out signify: multiple distinct compute nodes are failing with the same error, strongly pointing to a universal binary incompatibility across the system rather than an isolated issue on a single node. This pattern is a dead giveaway for an instruction set architecture problem.

The presence of .spack-env/view in your executable path is a huge clue. Spack is an amazing tool for managing complex software stacks in HPC, allowing for reproducible builds and handling dependencies. However, it requires careful configuration, especially regarding compiler flags and target architectures. When you build with Spack, you need to tell it exactly what CPU architecture you're targeting. Common pitfalls that lead to the 'illegal instruction' error include:

1. Compiling on a Login Node vs. Compute Node: Login nodes on HPC systems often have older, different, or more generic CPU architectures than the specialized compute nodes. If you compiled ACCESS-OM3 directly on a login node using a flag like -march=native, the compiler would optimize for the login node's specific CPU. When that executable is then moved to a compute node with a newer or different instruction set, it might try to use instructions not present on the compute node, or conversely, the compute node might not recognize instructions generated for the login node's older architecture if aggressive optimizations were used that weren't backward compatible. While -march=native is often tempting for perceived performance gains, it's a trap on heterogeneous systems or when compiling on a login node for compute node execution.
2. Incorrect Spack Target Specification: When building ACCESS-OM3 with Spack, you must explicitly specify the target architecture. For Setonix, this means telling Spack to build for target=zen3 or target=zen4 (or target=x86_64 with specific microarchitecture flags like skylake-avx512 if you are aiming for more general Intel/AMD compatibility, though targeting the specific AMD EPYC is usually better for peak performance). If Spack defaulted to a generic x86_64 without specific microarchitecture flags, or if it picked up an incorrect target, the resulting binary might not be optimally compiled or, worse, become fundamentally incompatible.
3. Missing or Incorrect Module Loads During Build: Even with Spack, the underlying system modules for compilers on Setonix (module load PrgEnv-aocc, module load gcc, etc.) need to be correctly loaded before you initiate your Spack build. An incorrect PrgEnv or an incompatible compiler version could lead to binaries that aren't fully compatible with the Setonix compute environment, or perhaps it used an older compiler without proper awareness of modern instruction sets. The specific combination of compiler (e.g., AOCC, which is AMD's optimized compiler, is often recommended for EPYC processors), MPI library (MPICH), and all scientific libraries (BLAS, LAPACK, NetCDF, HDF5) must all be built consistently for the target architecture. If, for example, a critical dependency like NetCDF was compiled with different optimization flags or for a different architecture than ACCESS-OM3 itself, the linker might pull in incompatible routines, leading to this very error in a linked library function.

In essence, your ACCESS-OM3 executable, access-om3-MOM6-CICE6, is likely speaking a dialect of machine code that the Setonix compute nodes simply don't understand because it was compiled for a slightly different CPU's capabilities. Rectifying this means revisiting your build process, particularly how Spack was configured to generate the executable for the specific AMD EPYC architecture present on the Setonix compute nodes. This architectural mismatch is the root cause you need to attack head-on to get your simulations running.

Practical Solutions: How to Fix This Beast!

Alright, team, now that we've pinpointed the 'why' behind that frustrating 'Illegal instruction (core dumped)' error plaguing your ACCESS-OM3 runs on Setonix Pawsey, it's time to switch gears to the 'how'. We're moving from diagnosis to active treatment, and I've got a comprehensive toolkit of practical solutions that HPC veterans routinely use to tackle these exact architectural mismatches. This isn't just about tweaking a setting; it's about fundamentally aligning your access-om3-MOM6-CICE6 executable with the specific instruction set capabilities of Setonix's powerful AMD EPYC processors. Remember, the core of the problem lies in your compiled code trying to execute an instruction the CPU doesn't recognize – a classic case of speaking different dialects of machine language. To resolve this, our strategies will focus on meticulous recompilation, rigorous environment verification, and, if absolutely necessary, forensic debugging. The goal is to ensure that every single instruction generated by your compiler for your complex ACCESS-OM3 climate model, including its MOM6 ocean and CICE6 sea ice components, is perfectly understood and executable by the Setonix hardware. This systematic approach is not only crucial for getting your current dev-MC_100km_jra_ryf configuration running, but it also equips you with essential knowledge for future projects on HPC systems. We’ll dive into the most probable and effective fix first: rebuilding your software with the correct architectural targets and compiler flags. Understanding these steps deeply will save you countless hours of troubleshooting down the line and allow you to harness the full potential of Setonix for your groundbreaking climate research. Let's get to it and turn this stumbling block into a stepping stone for seamless simulations!

The absolute most likely and most effective solution here is to recompile ACCESS-OM3 specifically for Setonix's CPU architecture. This is almost certainly your silver bullet, as the executable needs to be rebuilt with compiler flags that explicitly target the Setonix compute node CPUs. Since you're already leveraging Spack, a powerful package manager tailored for HPC environments, this process becomes much more manageable, but it still requires meticulous precision and attention to detail. First, you need to precisely identify the exact CPU architecture of the Setonix nodes you'll be using. Setonix predominantly features AMD EPYC processors, specifically a mix of Milan (Zen 3) and Genoa (Zen 4) generations. Knowing which generation you're targeting is crucial, as each has distinct instruction set capabilities. Pawsey documentation or a quick query to their support team can confirm the specific architecture of your allocated partition. Once identified, your Spack build command must incorporate the correct target directive, such as target=zen3 or target=zen4. If you omitted this target or relied on a generic x86_64 specification during your initial ACCESS-OM3 build, Spack might have made assumptions that ultimately led to the illegal instruction error you're encountering. Beyond the fundamental target, you might need to pass explicit compiler flags within your Spack build configuration. For instance, when using gcc, you could include flags='-march=zen3 -mtune=zen3' or -march=zen4 to ensure optimal performance and, more importantly, compatibility. However, a critical caveat here is to be extremely cautious with -march=native. While often recommended for maximizing performance on the specific machine it's compiled on, it becomes a detrimental trap in heterogeneous HPC systems where login nodes might have different CPU architectures than compute nodes. If you compiled with -march=native on a login node and then tried to run on a compute node with a different 'native' architecture, you'd immediately hit this instruction set mismatch. Therefore, always specify the target architecture directly, or, for maximum assurance, perform your Spack compilation within an interactive job (salloc) directly on a compute node, ensuring your build environment perfectly matches your runtime environment. Furthermore, leverage the recommended compiler suites available on Setonix. Pawsey often suggests specific programming environments, such as PrgEnv-aocc (AMD Optimizing C/C++ Compiler), which is specifically engineered to extract peak performance from AMD EPYC CPUs. Ensure your Spack environment is configured to use these recommended compilers and their associated MPI libraries (e.g., mpich). If your ACCESS-OM3 was built with gcc, verify that it's a version known to be compatible with Setonix's processor generation and that it's correctly loaded as a module. A typical robust Spack rebuild strategy would involve first establishing an interactive salloc session on a compute node, loading the appropriate system modules (e.g., module load PrgEnv-aocc, module load cray-mpich), configuring Spack to integrate these system compilers and MPI, cleaning any prior Spack build artifacts (spack clean --all), and then executing a command like spack install access-om3 %aocc@X.Y.Z ^mpich %aocc@X.Y.Z target=zen4 (remembering to adjust the compiler, version, and target as necessary). This meticulous rebuild process is, without a doubt, the most reliable method to eradicate architectural incompatibilities and get your ACCESS-OM3 running seamlessly.

Verify Module Loads (The Subtle Culprit)

Even if your Spack build is pristine, sometimes your runtime environment can cause issues. It’s crucial to ensure that the correct system modules are loaded consistently. During the build process, did you load the appropriate modules for the compiler, MPI library, and any other critical scientific libraries like NetCDF or HDF5 before you initiated your Spack build? Inconsistent module loads during the build can lead to parts of your software stack being compiled with different compilers or for different architectures, even if Spack attempts to manage dependencies. Similarly, during runtime, although your runom3.sh script doesn't explicitly module load any compilers or libraries (relying on the Spack view), if your Spack environment's view relies on specific system modules being available (which is common for MPI libraries and optimized math kernels), you need to ensure they are loaded. A spack module tcl refresh or similar command might be needed, or you might need to add module load commands to your runom3.sh if the Spack view isn't fully encapsulating all necessary runtime paths (e.g., for specific MPI or low-level libraries provided by the system).

Check Library Compatibility (The Dependency Web)

Your ACCESS-OM3 executable links against numerous libraries, including MPI, NetCDF, HDF5, and highly optimized numerical libraries like BLAS and LAPACK. An illegal instruction can sometimes originate not from your direct code, but from one of these dependencies if they were compiled for a different architecture than your main executable. To investigate, use ldd on your compiled executable (/software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6) to see all the shared libraries it links against. Carefully inspect these libraries to ensure they are also compatible with the target Setonix architecture. If a critical library (like a highly optimized BLAS/LAPACK routine) was accidentally linked from an incompatible build or an older system path, it could trigger the error. While Spack is excellent at handling dependencies by building them for the same target, a manual check can be insightful if recompilation alone doesn't immediately solve the problem.

Debugging with gdb (The Forensics)

The core dumped part of the error message is actually your friend here, even if it feels intimidating. A core dump is a snapshot of your program's memory and execution state at the precise moment of the crash. To use it, once your job crashes and produces a core file (it might be named core.<PID> or similar, found in your job directory), you can analyze it with the GNU Debugger (gdb). Load it using: gdb /software/projects/pawsey0889/cbull/setonix/2025.08/environments/om3/.spack-env/view/bin/access-om3-MOM6-CICE6 core.<PID>. Inside gdb, type bt (for backtrace). This will show you the sequence of function calls that led to the crash. Look for the lowest-level function call related to the illegal instruction. This can often pinpoint exactly which piece of code (or which library function) tried to execute the problematic instruction. While gdb can be complex, just getting the backtrace can give you valuable hints, sometimes even identifying the specific line of code in the source if you compiled with debug symbols (-g).

Simplify and Isolate (The Scientific Method)

If you're still banging your head against the wall, simplify your problem to isolate the cause. Can you run a much smaller ACCESS-OM3 test case? Perhaps a single node with minimal tasks? This helps rule out scaling issues or complex MPI interactions and focuses purely on the binary's ability to execute on a single node. Another good step is to compile and run a *simple MPI