Central to much of what Moab does is the concept of credentials. Credentials play a pivotal role in how many policies are not only defined, but actually carried out by the scheduler. They define how and by whom the system is accessed and used. One could even say they are at the heart of scheduler policy.
What Are the Credentials?
There are five credential types used by Moab:
- Quality of Service
I tend to divide this list into two different parts: traditional credentials and non-traditional credentials. This mental division really is used to keep what we normally consider to be a credential (e.g., a User) separate from that which we do not (e.g., a QoS).
|Traditional Credentials||Non-Traditional Credentials|
– Quality of Service
However, from Moab’s point of view, all of these five are used to do the same types of things in Moab, which will be discussed later in this article. At this point, the important thing to note is there are five credentials, which we will now discuss individually.
The User is the most conceptually simple of the credentials. All of us have at least one (though likely many, many) usernames. This is the most central and basic of credentials in Moab’s worldview. For example, a job may have an associated Account or Quality of Service, but it will always have a User credential associated with it.
In most cases, the User is based on the operating-system provided username and is mapped to the user’s UNIX UID. However, it is possible to provide this information to Moab using an Identity Manager. In an HPC system, this User credential is used when starting jobs, and so it must be valid on the underlying cluster.
Like the User credential, the Group credential is generally associated with the UNIX GID provided by the underlying operating system. However, it is possible for this also to provided to Moab by an Identity Manager. In most cases, effectively the Group and User become closely associated, as a particular username will be associated only with their primary group, as reported by the operating system. As the default for many Linux systems is to have the primary group name be the same as the user name, additional configuration of groups at the cluster level is needed to use the Group credential effectively.
Originally created for handling “budgetary accounts”, the Account credential can be thought of as a “project” or “bank account”. When used in conjunction with Moab Accounting Manager (MAM), Accounts represent pools of “money” or “credit limits” that member users can use to run jobs. A User can be a member of one or more accounts. When running workload, the User can associate a job with a particular Account, though a job does not require an association with an Account credential.
Beyond the “accounting” part of the Account credential, another interesting use is available. Because the Account basically allows the arbitrary grouping of Users into one or more “groups”, the Account credential can often substitute for the Group credential, as the Account credential does not have to directly tie to anything in the underlying operating system. In other words, administrators can use the Account credential to create arbitrary and/or transitory groupings of users (even without MAM installed). This powerful tool allows administrators the ability to better represent their organization within the system.
Moving on, we come to the first of the none-traditional credentials: the Class. Commonly known in HPC parlance as a “queue”, the Class credential is generally provided to Moab by one of the underlying workload resource managers (e.g., TORQUE). This represents different queues or buckets into which jobs can be placed while they wait to run. Each of these Classes (queues) can have different requirements about what type of job is or is not allowed to run using that credential. This configuration is generally done within Moab, as opposed to the resource manager(s).
There are a couple of different Class setups used on different clusters. The traditional approach is to have many different Classes (queues), each based on what types of jobs can be submitted to those Classes. For example, some sites have different Classes based on job priority (e.g., High, Medium and Low priority). Others split their Classes based on job size and duration (e.g., Small-and-Long, Small-and-Short, Big-and-Long and Big-and-Short). Some sites do a combination of these and other divisions. Alternatively, when using Moab, one can create a single Class (queue) and then use other policies to route and manage jobs.
The choice is left up to the system administrators to decide what is going to work best for them and their end-users.
That said, the Class is by far the most configurable of any of the credentials, providing the most options and power to the administrator for managing how workload flows through their cluster.
Quality of Service (Non-Traditional)
The final credential is the Quality of Service. This special credential is used to modify the rights, abilities and limits imposed by the other credentials. In other words, when used, it let’s exceptions be made to other policies. For example, a User may generally only be allowed to run 10 jobs in the system at any given time. However, if the User submits jobs with a special Quality of Service that allows 20 concurrent jobs, the User will be able to surpass their standard limit. Generally, this is associated with an additional cost to the User (otherwise, they’d use it all the time).
So, with that introduction, let’s look at some of the different ways credentials are used by Moab.
What Are Credentials Used For?
The short answer is: many things. Here we are going to only talk about the most common use cases.
When a User submits a job, that User “owns” the job. This means that user is able to run “administrative” actions on this job such as placing holds or canceling the job. No other User (with the exception of administrators) can perform these actions on jobs they do not own.
The same concept also exists for reservations, though, because only administrators are generally allowed to create reservations, the importance is much less apparent.
Access control is a major use for credentials, and is generally accomplished through the use of reservations. Any of the credential types can be used for defining access control into a reservation (or, effectively, to the nodes and other resources covered by the reservation). These can be mixed and matched, so it would be possible to create a reservation that says “only allow jobs submitted by users Carol, Bob, Jim or Suzie or using Account Physics or Quality of Service Expensive, except those jobs submitted using Class Short (reject those)”. Granted, that’s more complicated than normal, but it is possible.
This concept of using credentials for access control is central to Moab. A maintenance reservation is effectively a reservation with an empty ACL list, meaning no credential is allowed to start a job within the reservation. They are used to handle political conflicts and arguments over who gets access to what when in the cluster.
Many policies exist to configure the needed types of access control for nodes and other resource types.
One of the other ways credentials are often used is to set different types of limits on different end-users or groups of end-users. These limits often include limiting the number of jobs, processors, memory or other resources that can be used at any given time. As mentioned above, these limits can, in some cases, be overridden by the use of a Quality of Service credential.
As an interesting side note, there are some limits that do not affect the running of new jobs, but instead affect what queued jobs can receive priority or even the number of queued jobs for a particular credential. This can be enabled in order to combat different types of attempts to game the system (e.g., queue stuffing).
How Are They Configured?
Credentials are defined either in Moab’s configuration file or through the use of an Identity Manager. In the configuration file, credential configuration lines start with one of the following:
The credential’s name would go between the  braces. Configuration specific to that credential would then follow.
What’s This ADMINCFG Thing Then?
Since we are on the topic of the configuration file, one may notice the ADMINCFG lines. These are not directly related to defining credentials. However, they do define the administrative level of the listed User credentials. Normal end-users are set to Admin Level 5. Full-access administrators are Admin Level 1. The other levels in between represent different levels of access, though most systems only use levels 1 and 5.
As an additional side note, the first User listed as ADMINCFG is the user used to run the Moab server daemon.
This represents a very quick introduction to Moab’s credential system. Consider doing one or both of the following: