Trev Harmon

  • Home
  • Business
  • HPC
  • Cloud
  • Big Data
You are here: Home / Cloud / Understanding Moab Scheduling: Part III

March 10, 2014 by Trev Harmon Leave a Comment (REPOST)
March 3, 2014 at Adaptive Computing (ORIGINAL)

Understanding Moab Scheduling: Part III

This entry is part 3 of 3 in the series Understanding Moab Scheduling

Understanding Moab Scheduling
  • Understanding Moab Scheduling: Part I
  • Understanding Moab Scheduling: Part II
  • Understanding Moab Scheduling: Part III

In the final installment of this series, I’m going to give two step-by-step examples of how Moab actually schedules jobs. The first will show a fairly standard scenario, while the second will include preemption. If you don’t recall how the Moab scheduling cycle works, you may want to do a quick review before proceeding.

Example: Basic Scheduling

1.

Here on the right we have a basic cluster usage over time graph. The y-axis represents the nodes in the system and the x-axis represents time. Blocks in the field represent how the nodes are being used at a specific time. Grey boxes are reservations, while blue, red and pink boxes represent jobs. On the left side, one will find the job queue, with jobs sorted from highest to lowest priority. The relative “height” of each job represents how many nodes it requires. Three of these nodes (the top three red ones) are special. They have a node-locked software license that is required by all red and pink jobs.As jobs are moved from the queue to a running state, they will be removed from the queue and placed on the field. A vertical “NOW” bar will keep us informed on our progress along the x-axis. Clicking on the image will open a new tab with a zoomed-in copy of the image. example-scheduling-01

2.

Here we are at the scheduling phase of the first scheduling iteration. Moab has taken the top two priority jobs and started them. Notice they’ve been removed from the queue. The third highest priority job did not fit, so a reservation has been made for it in the future where it will fit. By default, this is only done for the top priority job that cannot start. However, if you desire a priority reservation to be made for more than one job, you can increase RESERVATIONDEPTHThe priority reservation(s) will be re-evaluated (and destroyed/re-created) every scheduling iteration in order to take advantage of updated information. example-scheduling-02

3.

Moab now switches to backfill mode and continues down the job queue until it finds a job that will be able to start and won’t disturb the priority reservation made for the highest priority queued job. example-scheduling-03

4.

Moving on through time, Moab gets to the next location where it can start a job. Checking the highest priority job, it is found it will not fit within the available space. Its reservation is updated, but left where it is.Switching to backfill mode, Moab finds a job in the queue that will be able to start. It does not select one of the red or pink jobs, even though they are small enough to fit in the available space, because they require one of the “red” nodes at the top of the cluster. example-scheduling-04

5.

Reaching the next point in time where jobs have finished and thereby freed space in the cluster, we find we still are only scheduling in backfill mode because the highest priority job just can’t fit in the available space.The highest priority pink job is started in the “red” nodes. The red and other pink job then must wait because all “red” nodes are now in use. Other jobs are pulled from the queue in priority order to fill the remaining space. example-scheduling-05

6.

This process repeats itself over and over again. Let’s fast-forward.Fast Forward >> example-scheduling-06

7.

Fast Forward >> example-scheduling-07

8.

Fast Forward >> example-scheduling-08

9.

Fast Forward >> example-scheduling-09

10.

Fast Forward >> example-scheduling-10

11.

Now, at this point we get to something that is interesting. The highest priority job in the queue has been waiting for some time. However, it is now its turn to start. So, it’s moved from the queue and starts to run.Moab then continues down the queue trying to start the next highest priority job, but immediately discovers there is no room to start it. So, Moab creates a reservation for it out in the future, and leaves it in the queue.Moab then switches to backfill mode, but as there are no more idle jobs in the queue, it finishes scheduling for that iteration. example-scheduling-11

12.

When we finally reach the reservation for the last job, the final job removes that reservation and starts. example-scheduling-12

Example: Preemption Scheduling

1.

Now we will take a look at the same process, except we will have a high priority job preempt some of the running workload, which will be re-queued.We start with basically the same cluster set-up that we used in the last example, though some of the job sizes and ordering have changed slightly. example-preemption-01

2.

Moab starts the three highest priority jobs. The fourth, a pink job, is given a priority reservation. Moab switches to backfill mode, but finds there is no room left in the cluster to run jobs at this given moment. example-preemption-02

3.

As Moab is running the jobs, a high priority job (the gold one) is submitted to the cluster. This job is part of a QoS that allows it to preempt other workload on the system. It’s a very important job. example-preemption-03

4.

Moab preempts the lowest priority job that is currently running in the system that is also preemptable. In this case, we have told Moab to re-queue preempted jobs, if possible. It’s placed back in the queue. example-preemption-04

5.

However, the preempted job, not the pink one, is now the highest priority queued job. So, Moab removes the priority reservation for the pink job, and creates one for the preempted blue job. It then switches to backfill mode, but finds on available space to start any backfill jobs. example-preemption-05

6.

When we reach the point where Moab again has room to start new jobs, it starts the two highest priority jobs in the queue before it runs out of space. It then creates a priority reservation for the next highest priority job in the queue (the red one). Switching over to backfill mode, Moab again finds no available space for backfill jobs. example-preemption-06

Conclusion

Hopefully, this series has helped you understand how Moab scheduling works. For additional information please consult our documentation, take our administrator training course, or visit us at MoabCon, where many such topics are discussed and you’ll have access to talk directly with Moab engineers.

Series Navigation<< Understanding Moab Scheduling: Part II

Filed Under: Cloud, HPC Tagged With: backfill, Cloud, HPC, job, Moab, preemption, qos, reservations, scheduling, scheduling cycle, workload

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

The Manifesto

Conscious Business Ethics

BusinessEthics300

Learn why it matters. Then sign it!

Get Updates!

Affiliate Disclosure

This site offers affiliate links to some online retailers such as Amazon.com in conjunction with hyper-linked books, movies, music and other such items. IF you click on these links and subsequently make a purchase, I will receive a small percentage of the transaction price

Twitter: trev_harmon

  • RT @Calvinn_Hobbes: 33 years ago today (November 18, 1985), one unique tiger was captured by a precocious boy using a tuna sandwich. That w… 12:21:10 PM December 18, 2018 ReplyRetweetFavorite
  • We have some exciting opportunities at @evernym in the #informationtechnology, #DevOps and #SaaS/#Cloud spaces. Hel… https://t.co/Yrn2PgKV8d 06:36:58 PM May 15, 2018 ReplyRetweetFavorite
  • A memorable end to a series of memorable moments in a memorable conference — Elevate Humanity through Business… https://t.co/WmuYuxYYXN 12:42:35 PM May 02, 2018 ReplyRetweetFavorite
  • “When you don’t have resources, someone else chooses your destiny for you.” — @jeromedlove at #CCAC2018 12:50:54 PM May 01, 2018 ReplyRetweetFavorite
  • RT @ConsciousCap: “You cannot be a Conscious Leader if you are unwell” - @DrEvaSelhub on stress and the effect on overall wellness. #CCAC20… 12:42:56 PM May 01, 2018 ReplyRetweetFavorite
  • RT @ConsciousCap: Making the world a better place through PB&Js, an idea born from a #ConsciousCapitalism conference by @jeffsinelli #CCAC2… 07:56:27 AM May 01, 2018 ReplyRetweetFavorite
  • “When you find what you believe, it changes everything.” — @jeffsinelli @ #CCAC2018 07:55:09 AM May 01, 2018 ReplyRetweetFavorite
  • “What happens inside us affects what happens between us.” — @michellebkinder @ #CCAC2018 06:53:11 AM May 01, 2018 ReplyRetweetFavorite
  • Michelle Kinder is talking about how until the world is as good for #children as it is for #business, business won’… https://t.co/FznhhhqZ7S 06:48:32 AM May 01, 2018 ReplyRetweetFavorite
@trev_harmon

Writer, software architect, educator, blogger, photographer, would-be designer, and a believer in the power of simplicity and human-based design.

Other Blogs

Trev Harmon can also be read at:

  • Dream.Learn.Discover
    Primary Author -- This blog is about seeing the good in the world. With all the bad, evil and destruction, there are many, many people who are creating good in their sphere of influence. Some of these spheres are large and some are small. There is a time allotted to each one of us. It is with this time some decide to do remarkable things, though they may not believe them to be remarkable at the time.
  • Adaptive Computing
    Contributor -- The world of high-performance, cloud and supercomputing is opening the way for many new and exciting discoveries. As we push our quest for knowledge forward, technology will play a key role in supplementing our ability to learn and discover.

Copyright © 2019 · Executive Pro Theme on Genesis Framework · WordPress · Log in