Understanding Moab Scheduling: Part III

In the final installment of this series, I’m going to give two step-by-step examples of how Moab actually schedules jobs. The first will show a fairly standard scenario, while the second will include preemption. If you don’t recall how the Moab scheduling cycle works, you may want to do a quick review before proceeding.

Example: Basic Scheduling
1.	Here on the right we have a basic cluster usage over time graph. The y-axis represents the nodes in the system and the x-axis represents time. Blocks in the field represent how the nodes are being used at a specific time. Grey boxes are reservations, while blue, red and pink boxes represent jobs. On the left side, one will find the job queue, with jobs sorted from highest to lowest priority. The relative “height” of each job represents how many nodes it requires. Three of these nodes (the top three red ones) are special. They have a node-locked software license that is required by all red and pink jobs.As jobs are moved from the queue to a running state, they will be removed from the queue and placed on the field. A vertical “NOW” bar will keep us informed on our progress along the x-axis. Clicking on the image will open a new tab with a zoomed-in copy of the image.
2.	Here we are at the scheduling phase of the first scheduling iteration. Moab has taken the top two priority jobs and started them. Notice they’ve been removed from the queue. The third highest priority job did not fit, so a reservation has been made for it in the future where it will fit. By default, this is only done for the top priority job that cannot start. However, if you desire a priority reservation to be made for more than one job, you can increase RESERVATIONDEPTHThe priority reservation(s) will be re-evaluated (and destroyed/re-created) every scheduling iteration in order to take advantage of updated information.
3.	Moab now switches to backfill mode and continues down the job queue until it finds a job that will be able to start and won’t disturb the priority reservation made for the highest priority queued job.
4.	Moving on through time, Moab gets to the next location where it can start a job. Checking the highest priority job, it is found it will not fit within the available space. Its reservation is updated, but left where it is.Switching to backfill mode, Moab finds a job in the queue that will be able to start. It does not select one of the red or pink jobs, even though they are small enough to fit in the available space, because they require one of the “red” nodes at the top of the cluster.
5.	Reaching the next point in time where jobs have finished and thereby freed space in the cluster, we find we still are only scheduling in backfill mode because the highest priority job just can’t fit in the available space.The highest priority pink job is started in the “red” nodes. The red and other pink job then must wait because all “red” nodes are now in use. Other jobs are pulled from the queue in priority order to fill the remaining space.
6.	This process repeats itself over and over again. Let’s fast-forward.Fast Forward >>
7.	Fast Forward >>
8.	Fast Forward >>
9.	Fast Forward >>
10.	Fast Forward >>
11.	Now, at this point we get to something that is interesting. The highest priority job in the queue has been waiting for some time. However, it is now its turn to start. So, it’s moved from the queue and starts to run.Moab then continues down the queue trying to start the next highest priority job, but immediately discovers there is no room to start it. So, Moab creates a reservation for it out in the future, and leaves it in the queue.Moab then switches to backfill mode, but as there are no more idle jobs in the queue, it finishes scheduling for that iteration.
12.	When we finally reach the reservation for the last job, the final job removes that reservation and starts.
Example: Preemption Scheduling
1.	Now we will take a look at the same process, except we will have a high priority job preempt some of the running workload, which will be re-queued.We start with basically the same cluster set-up that we used in the last example, though some of the job sizes and ordering have changed slightly.
2.	Moab starts the three highest priority jobs. The fourth, a pink job, is given a priority reservation. Moab switches to backfill mode, but finds there is no room left in the cluster to run jobs at this given moment.
3.	As Moab is running the jobs, a high priority job (the gold one) is submitted to the cluster. This job is part of a QoS that allows it to preempt other workload on the system. It’s a very important job.
4.	Moab preempts the lowest priority job that is currently running in the system that is also preemptable. In this case, we have told Moab to re-queue preempted jobs, if possible. It’s placed back in the queue.
5.	However, the preempted job, not the pink one, is now the highest priority queued job. So, Moab removes the priority reservation for the pink job, and creates one for the preempted blue job. It then switches to backfill mode, but finds on available space to start any backfill jobs.
6.	When we reach the point where Moab again has room to start new jobs, it starts the two highest priority jobs in the queue before it runs out of space. It then creates a priority reservation for the next highest priority job in the queue (the red one). Switching over to backfill mode, Moab again finds no available space for backfill jobs.

Conclusion

Hopefully, this series has helped you understand how Moab scheduling works. For additional information please consult our documentation, take our administrator training course, or visit us at MoabCon, where many such topics are discussed and you’ll have access to talk directly with Moab engineers.