Over the last few weeks, I have been involved in discussion with various external parties about Ceetron’s visualization solutions and how they can be applied in the HPC world. I thought it could be interesting to explore this issue in more depth, with some assistance from my colleague Andres Rodriguez-Villa.
Ceetron has for many years had a presence within the HPC community, but primarily as bolted-on visualization solutions to various HPC clusters. Examples of these are the Norwegian Metacenter for Computational Science at NTNU, Trondheim; Statoil HPC Cluster, Stavanger; and GE Aviation HPC Lab, India. In general, Ceetron has traditionally had its core customer base in industries that are heavy consumers of CPU power, including oil and gas and automotive.
Methodologically, we decided to approach the above issues through a structured interview with Dr. Anne Cathrine Elster at HPC-Lab at NTNU, Trondheim. The interview was conducted on 21.09.2015. For a bio for Dr. Elster, see below. What follows is a (heavily) edited transcript of our conversation:
CI: Good afternoon Dr. Elster, could you let us know a bit more about HPC-Lab?
HPC-Lab is now an acronym for Heterogeneous and Parallel Computing Lab. Our lab includes a five-node GPU-accelerated ccNUMA cluster from NumaScale, 15+ workstations with NVIDIA GTX 980 graphics cards, and an 85-inch Samsung 4K Professional Display QM85D. We are also about to receive a donation from Maxeler and Chevron (MAX3 box), as part of the EU project we have, which again is related to the work we are doing for the oil and gas industry. We are also an NVIDIA GPU Research and Teaching Center, and are enthusiastic owners of 1 NVIDIA Tesla K40 and 2 Tesla K20, as well as several SMD GPUs and low-power toolkits such as the NVIDIA TK1 and Adapteva’s Parallella cards.
Our research focuses on tools for high-end and/or low-powered heterogeneous and parallel processing typically with GPU accelerators.
CI: I am aware of your historical collaboration with Ceetron’s CEO, Tor Helge Hansen. Could you give us some context for this collaboration?
Dr. Elster: Tor Helge has been an external examiner for a large number of my MS students. This has given us an understanding of the software industry’s assessment of relevance, and I believe for Tor Helge and Ceetron have resulted in a deep insight into trends and upcoming themes in the wider field of HPC and visualization for HPC.
CI: In addition to your work on GPU-based architectures, you have published extensively on parallelizing and optimizing applications for medical image processing. What are, from a software- and hardware-architectural view, the fundamental differences between image processing and simulation?
Dr. Elster: There is a lot of overlap from a computing point of view, and in HPC-Lab we do work both on image-processing applications, like in medical imaging, and on simulation applications, like seismic imaging and reservoir simulation. However, medical image-processing applications often come with real-time constraints, which is a less common hard requirement when doing general simulations.
CI: Could you say something about your research interests today?
Dr. Elster: We have a very interesting EU project, Cloud Lightning, on self-organizing self-optimizing clouds for HPC. It is a H2020 project. We are furthermore doing significant work with Statoil, especially on the OPM project, which is an open source project. We have close collaboration with SINTEF and NTNU, specifically with Frank Lindseth in SINTEF Medical Technology and Bjørn Angelsen in NTNU’s Group for Medical Imaging.
CI: Let us then get to the core of the matter: What are the big trends in HPC community today, and how may these trends impact the space of visualization technologies for HPC?
Dr. Elster: Well, it is all about hardware roadmaps, as exemplified by Intel’s Knights Landing, and NVIDIA’s NVLINK and their collaborations with IBM.
Then it is about multi-level modelling and coupled multi-model simulation. This will greatly increase model sizes, and ultimately requirements to computational performance.
I also think we will see progress in the area of computational steering, and the ability to steer computations towards specific regions of interest or hotspots during the actual simulation. Such ability will require effective visual manipulation tools as well as effective controls, and the ability to process and visualize in real time, and with no latency.
CI: Today everybody is talking about clouds. At the same time, we see the emergence of internet of things. Where will we see central storage / processing, and where will we see local storage / processing? Will there in the future be a space for traditional on-premise storage and processing?
Dr. Elster: I think we will see all paradigms coexist in the future, including the high-performance workstation for engineering applications under the CAE professional’s personal desk, but also with access to at least private cloud resources. On the other hand, real-time medical imaging, for example, may process data locally on the probes as well as on smaller hand-held devices such as tablets and or high-end portables.
CI: If we dig further into the particular issue of central / cloud-based rendering vs. device-based rendering, which paradigm will in the future dominate and why, on what devices?
Dr. Elster: It depends, but it is a fair observation that many smaller devices have high-performance GPUs that may be used for local rendering.
CI: If we restrict ourselves to CFD and FEA, we can see that HPC-based visualization, as opposed to visualization for HPC, is being used for high-end computations in the research community. Do you think that in the near future this trend will become mainstream in industrial applications?
Dr. Elster: If we take a look at what is going on outside the CAE community, this is certainly the case, and Pixar and animation movie production in general come to mind. Inside the CAE community, I guess that real-time requirements for example associated with computational steering, increasing model sizes / multi-level modelling, and requirements to visual quality / more sophisticated visual modelling will drive the computational requirements beyond what can be obtained by a bolted-on visualization solution running on a single CPU.
CI: And what devices will in the future be used for visualization for typical workflows, say by a CAE professional or a reservoir engineer?
Dr. Elster: It depends on the use case. We will certainty see novel or developments of existing devices, for example in the form of various VR devices like Oculus Rift (we have such a device in our lab), various types of immersive caves (like the ones used by reservoir engineers), and plainly very large / very high-resolution screens. And, then we have mobile devices in the form of pads and large phones, for typical mobility-enabled use cases. An interesting emerging technology there is the VR gear that integrates Oculus Rift with Smartphones. https://www.oculus.com/en-us/gear-vr/ . We are also seeing holographic imaging becoming more and more accessible.
Thanks to Dr. Elster for her insightful contributions.
We also decided to discuss Dr. Elster’s perspectives with Fredrik Viken, CTO of Ceetron. Here are his comments, after having read the above transcript: “First, Ceetron focuses on a different part of the total stack from that of HPC-Lab, as we offer our 3D Components product to CAE tool providers with applications deployable on an HPC infrastructure, and viewers for end users wanting to distribute, share, and collaborate on models created on HPC clusters.
In general, I share Dr. Elster’s general perspectives. We observe in the community a Moore’s type law regarding typical model sizes in the industry, probably due to advances in multi-physics modelling and coupled problem simulations and generally multi-level, multi-domain, multi-scale, which makes HPC-type clusters more and more part of the standard tool box for the CAE professional with a PhD in CFD or FEA. This observation has made it imperative for Ceetron and Ceetron’s OEM customer to create offerings that can be deployed on HPC infrastructures.
Regarding the question about local rendering, and just as an example, IPad Pro outperforms 90% of the laptops produced in 2015 when considering GPU power. The GPU power of phones and tablet is growing faster than Moore’s law, and is one of the main reasons for Ceetron pursuing the 3D streaming / local rendering paradigm.
One challenge for HPC environments, at least for use cases based on on-the-fly viewing, for example computational steering, is of course that any decent HPC solver will be storing results in a distributed way, and thus that the model will not fit in the memory of a single machine. Distributed visualization is an imperative for HPC / cloud computations, as the result sets might be in the 100GB / 1TB range, which is not possible to transfer to the client in a short time, and the client would be unable to process it. The cluster/cloud has storage/memory/compute power to handle this, and using smart adaptive 3D streaming, our goal is to be able to visualize any model on any device.
Another challenge is that HPC environments tend to be very optimized, meaning very specific hardware-wise (and challenging to create general high-performance software for, at least within the business logic of licensed standardized software). It is a fair guess that we in the future will see 3D Components, HPC Edition. To get there, we would be looking for attractive collaboration opportunities with credible HPC initiatives.”
But going back to the key objectives of this blog post: My personal takeaways from the interview with Dr. Elster were threefold: i) we (meaning the complete value chain: Ceetron, OEMs in CFD and FEA space, and CAE professionals) need to prepare for a future world in which data storage and processing will happen in centralized compute clusters, whether we call it cloud or HPC clusters; and ii) we must create solutions to visualize on local devices not tightly connected to these compute clusters; and iii) the traditional CPU-based architecture may in the future be replaced with more heterogeneous architectures, including GPU-based architectures.
Bio for Dr. Anne Cathrine Elster: Dr. Elster is Associate Professor of high-performance computing in Dept. of Computer & Information Science at the Norwegian University of Science and Technology (NTNU) in Trondheim, Norway, where she is heading the HPC-Lab (Heterogeneous and Parallel Computing Lab). She is also Visiting Fellow at ICES/University of Texas at Austin, USA. Her research interests are in parallel and heterogeneous computing, including GPU computing, and tools for program optimization. She started her research career at Cornell where she got her MS in 1988 and PhD in 1994. She is a senior member of IEEE, a CUDA book author, and has advised a large number of Post Docs, PhD students and Master students.