As I mentioned last week, this afternoon I was able to speak to a wonderful audience at the SC13 Exhibitor Forum. This particular track at the conference is rather interesting, as it really is put on as an opportunity for vendors to pretty much brag about what they are doing. There’s the perfunctory nod to the industry, and then onto what they are doing. And, there’s nothing wrong with this. Vendors are just as much a part of the HPC community as the cluster owners and users. All of us are needed in order for the ecosystem to remain happy.
I took this talk as an opportunity to not only look at where we are going, but to see from where we have come. Supercomputing is a rather interesting experience. There are thousands who gather for the week. We spend much of it discussing what is wrong, what problems need to be solved and how different groups are attempting to address the issues.
It can be a bit taxing.
Regardless of what we do or how far we progress, there are always new challenges that are waiting in the wings to jump us once we deal with the current lot. It feels like we can never truly win. However, one thing this industry has that I absolutely love is best summed up in a quote from one of my favorite bands, Counting Crows:
If you never stare off into the distance, then your life is a shame.
~ Counting Crows ‘Mrs. Potters Lullaby‘
There is that constant looking forward to the future, with a belief we can overcome the problems. I remember when we first reached peta-scale. It was an exciting day, but we immediately turned our sights on the next big target, exa-scale. At that time, I remember reading in one of the industry rags (I wish I could remember which) a brief article outlining what it would take to build an exa-scale system with the technology of the day. Here are some of the attributes the article mentioned:
- Power requirements of a medium-size city
- <blah> <blah> <blah>
- Mean Time Between Failure: 18 seconds
Really, it was that last one that caught my attention (obvious, as I can’t really remember any of the other specifics). So much hardware would be required to run the system that a failure with at least one part would occur every 18 seconds. Good luck getting a clean run of Linpack. Additionally, I can just imagine the army of minions required to constantly be running through the computer room swapping out bad components.
Now, we may not make it to this next big milestone by 2018. I don’t know. We’ll have to wait and see. But, it’s important to look back at where we’ve come over the last half century. Sure, there have been many mistakes, false starts, and silly decisions, but on the whole our industry has moved steadily forward.
I believe we are on the cusp of some miraculous changes. Granted, in some ways this does come from anecdotal evidence, but I see it nonetheless. Let’s look at a couple.
Over the years, I’ve had many discussions with different industries that are effectively doing HPC work, but using a different nomenclature. Conversations have gone thus:
With the advent of Big Data, I’m finding I’m not having these types of conversations nearly so frequently. It’s as if we’re all starting to see things simply as different forms of computing.
In my talk this afternoon, I put forward the idea that all of these different silos of computing (HPC, Big Data, Cloud, Scientific Computing, etc.) really do need to be thought of simply as Computing. I know it’s a gross oversimplification, but I find it to be useful.
At the end of the day, the thing that truly matters is whether or not the user is enabled to solve their problems. It’s really not about the hardware vendors, the software vendors or even the administrators. It’s really about whether or not users are able to solve their problems, whether it’s understanding the forces on a bit of structural steel or solving the greatest issues facing humanity today.
Convergence to Workflow
Another major theme I’m taking away with me from the conference this year, is this idea of convergence to workflow. It was another major theme in my talk, but it wasn’t until I got here and was able to spend time looking around at what others were doing that I discovered how prevalent it had become.
I was surprised.
Let’s take the most basic case, when we do calculations in an HPC job and then hand things over to a visualization system provided on the cluster, we’ve created a basic workflow. In my talk, I contended as we move forward these workflows will become more and more complex, pulling in things from Big Data, Cloud Computing, etc. And, people are truly interested in this.
Two or three years ago, I would have not have been taken seriously had I mentioned virtualization and HPC in the same sentence. Two years ago, Cloud and HPC together in a comment would have elicited the same reaction. They were separate concepts and never the twain should meet. This year, I saw these pairings all over the place, in talks, abstracts, classes and vendor presentations. These pairings represent convergence.
Another important convergence was our announcement with Intel around easily enabling Hadoop to run efficiently on HPC clusters. This is a big deal, not only as it will fill the gap left by the apparent abandonment of Hadoop on Demand by the project, but also because it represents the convergence of the HPC and Big Data silos that is occurring.
Convergence is happening all over the place, and with it we will build workflows.
[The computer utility] represents a natural extension of familiar technology in such a way as to create a revolution in the distribution and utilization of computer power, a revolution that promises ultimately to make that power as freely available as is the telephone service of today. Out of this availability will flow social changes and opportunities for human development that promise to make the last decades of the twentieth century among the most critical that the human species has had to face during its million-odd years of existence. The present state of the computer art is advanced enough to permit the immediate construction of many extremely useful types of computer utility, but the full exploitation of the concept will obviously require major advances in the entire field of computer technology.
We are inheritors of a great vision, which we are on the cusp of seeing fulfilled.
Join us on the journey.