The raindrop again but with 8 billion (!) cells in 512GB GPU memory @ProjectPhysX

Uploaded January 2023 | Updated September 2026, 2 weeks ago

I have now sucessfully extended FluidX3D to multi-GPU, allowing to pool VRAM of multiple GPUs installed in a server node via domain decomposition, here 8x AMD Instinct MI200 64GB. The multi-GPU update is now published on GitHub: github.com/ProjectPhysX/FluidX3D

This is an 8 billion grid cell simulation of a d = 7 mm diameter terminal velocity (9.55 m/s) raindrop impact at 20° inclination, simulated for 3 milliseconds (34650 LBM time steps). The box is 5d x 5d x 4.25d and the pool height is 2d. For reproducibility: The dimensionless numbers for this setup are Re = 63618, We = 8864, Fr = 36.44, Ca = 0.1393, Bo = 6.674.
Velocity, density and fluid mass are solved at 8 billion lattice points (2122 x 2122 x 1804) with D3Q19 SRT and FP16S memory compression. The simulation requires 512 GB video memory and was conducted on 8x AMD Instinct MI200 GPUs at Jülich Supercomputing Centre (JSC) JURECA-DC-MI200. Combined compute+render time was 330 minutes. 4K60 rendering is done via domain decomposition rasterization, where each GPU renders only its own domain and the frames are seamlessly overlayed via their z-buffers. 4x 451 frames (6.3GB) are generated.

Find physical accuracy validation of the raindrop setup in this paper: doi.org/10.1186/s43591-021-00018-8

How is it possible to squeeze 8 billion grid points in only 512GB VRAM?
I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 67 Bytes/node for D3Q19 LBM with the Volume-of-Fluid extension, or 1/3 of conventional codes:

1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.
Paper: doi.org/10.3390/computation10060092

2. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.
Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formats

Graphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data ever has to be copied to the CPU or hard drive, but only rendered frames instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.
Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCL

Timestamps:
0:00 front view
0:07 side view
0:15 back view
0:22 top view

#CFD #GPU #FluidX3D #OpenCL

The raindrop again but with 8 billion (!) cells in 512GB GPU memory

PhysX3D [GRAVITY SIMULATION] Ulysses slingshot on November 4th 2098

$Combined scientific CFD simulation and interactive raytracing with OpenCL (IWOCL & SYCLcon 2022) This is my IWOCL & SYCLcon 2022 technical talk. Extended Abstract: https://www.researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCL Slides: https://www.iwocl.org/wp-content/uploads/06-presentation-iwocl-syclcon-2022-lehmann.pdf 0:00 Intro & Demos 4:16 Basic Idea 6:54 Lattice Boltzmann CFD 8:35 Benchmarks & Efficiency 11:33 Application 12:37 Graphics with OpenCL 23:06 Conclusions 24:16 More Demos!! 26:09 Outro One of the main uses for OpenCL is compute applications where rendering is done externally. However separating simulation and rendering has many disadvantages, especially the slowdown caused by copying data from device to host, and needing to store raw data on the hard drive. A much faster approach is to implement both simulation and rendering in OpenCL. The rendering kernels have direct read-only access to the raw simulation data in ultra-fast GPU memory. This eliminates almost all PCIe data transfer, allowing for interactive visualization in real time while the simulation is running. Although OpenCL does not have existing functionality for graphical rendering, being a general compute language, it allows for implementing an entire graphics engine. On top, specific low-level optimizations make this OpenCL graphics engine outperform any existing rendering solution for this scenario, enabling fluid raytracing in real time on even non-RTX GPUs. This is demonstrated with the software FluidX3D [1]. First I introduce the lattice Boltzmann method for simulating physically accurate fluid flow and discuss optimizations: Being a memory-bound algorithm, coalesced memory access is key. This is achieved through AoS data layout and the one-step-pull LBM streaming scheme. One-step-pull leverages that misaligned read penalty is much smaller than misaligned write penalty on most GPUs. Roofline analysis shows that with these optimizations, the LBM runs at 100% efficiency on the fastest GPUs [2]. For free surface flows, the LBM is extended with an efficient Volume-of-Fluid (VoF) implementation [3]. This allows covering new grounds in science: FluidX3D has been used to simulate more than 1600 raindrop impacts to evaluate how microplastics transition from the ocean surface into the atmosphere [4]. Rasterization on the GPU is parallelized not over pixels but lines/triangles instead. Each line/triangle is transformed from 3D to 2D screen coordinates and then rasterized onto the frame (integer array) with Bresenham [5] and z-buffer. The raytracing graphics are based on a combination of fast ray-grid traversal and marching-cubes, leveraging that the computational grid from the LBM already is an ideal acceleration structure for raytracing. The idea is simple: Through each pixel on screen, shoot a reverse light ray out of the camera and see where it intersects with a surface in the scene. Then (recursively) calculate reflected/refracted rays and mix colors. If a ray doesnt intersect anything, its color is determined by the skybox. With mesh surfaces consisting of triangles, computation time quickly becomes a problem, as all triangles have to be tested for intersection. To overcome this, an acceleration structure is required. While games often use a bounding volume hierarchy, the LBM already provides an ideal alternative acceleration structure: the simulation grid. The corresponding algorithm is called ray-grid traversal: When a ray shoots through the 3D grid, intersections with the surface only have to be checked for at each traversed grid cell rather than the entire grid. Triangles are generated on-the-fly with marching-cubes and ray-triangle intersections are checked with Möller-Trumbore. If an intersection has been found, the normals are calculated on the 8 grid points spanning the cell, and are trilinearly interpolated to the intersection. The interpolated surface normal makes the raytraced surface appear smooth. On the GPU, the ray(s) for each pixel are computed in parallel. It is important how to align the OpenCL workgroups: best performance is achieved for 8x8 pixel tiles; this is about 50% faster than 64x1 tiles, because all rays of the workgroup are more likely to traverse the same grid cells, improving memory broadcasting. For marching-cubes, the algorithm by Paul Bourke [6] is implemented in OpenCL. Table size is reduced to 1/8. The Möller-Trumbore algorithm [7] is implemented branchless. This raytracing implementation is fast enough to run in real time for even the largest lattice dimensions that fit into GPU memory. This is demonstrated on the most realistic simulation of an impacting raindrop ever done [8]. [1] https://doi.org/10.15495/EPub_UBT_00005400 [2] https://arxiv.org/abs/2112.08926 [3] https://doi.org/10.3390/computation10020021 [4] https://doi.org/10.1186/s43591-021-00018-8 [5] https://doi.org/10.1147/sj.41.0025 [6] http://paulbourke.net/geometry/polygonise/ [7] https://doi.org/10.1080/10867651.1997.10487468 [8] https://youtu.be/1g_zFsvScME$

Visualizing 144 TeraByte of CFD data for fun

hw-smi - a minimal, cross-compatible CPU/GPU telemetry monitor

PhysX3D [GRAVITY SIMULATION] Extreme Performance

$From Boxes to photorealistic Fluid (marching-cubes + raytracing) From Boxes to photorealistic Fluid - how does ot work? The fluid simulation is done with the lattice Boltzmann method (LBM) with the Volume-of-Fluid (VoF) extension and piecewise linear interface construction (PLIC) for surface tension. VoF-LBM: https://www.researchgate.net/publication/349552856_High_Performance_Free_Surface_LBM_on_GPUs PLIC: https://www.mdpi.com/2079-3197/10/2/21 VoF-LBM provides a 3D grid with fluid fill levels, used as isovalues for marching-cubes, which generates a contiguous triangle mesh representing the water surface. Marching-cubes: http://paulbourke.net/geometry/polygonise/ With raytracing I shoot light rays from the camera through the 3D isogrid. For each grid cell that a ray traverses, I run marching-cubes to generate the triangles on-the-fly, then check for ray-triangle intersections with any of these triangles. If the ray intersects, I dont use the flat-face-surface nromal of the triangle, as this would lead to visual artefacts on the reflected/refracted rays. Instead, I interpolate the surface normal at the intersection point in order to have a smoothly appearing surface. How does surface normal interpolation work with marching-cubes? I get the normals directly at the lattice points with gradient (central derivative) on the isogrid, then linearly interpolate to get the normals at triangle vertices and linearly interpolate again to where the ray intersects the triangle with Barycentric coordinates on the triangle. The 3D lattice that the LBM provides already is one of the ideal acceleration structures for raytracing. Any ray only has to check for possible intersections with iso-surface triangles contained within the traversed grid cells. This makes it - when the lattice is not too large - even run in real time on non-RTX GPUs. New video with surface normal interpolation: https://youtu.be/UUYrWUpPQwY Old video without surface normal interpolation: https://youtu.be/9Rfu9ZqW894 The simulation setup you see here is a physically accurate 4mm diameter terminal velocity raindrop impact in water, although at rather low 256x256x218 resolution for this real time demo. See my published research paper in Microplastics and Nanoplastics, where the simulation is validated to great extent: https://microplastics.springeropen.com/articles/10.1186/s43591-021-00018-8 This simulation as well as the raytracing graphics are running in real time on a single Nvidia Titan Xp GPU. The simulation software used is FluidX3D, an OpenCL implementation of the lattice Boltzmann method. FluidX3D runs at peak hardware efficiency on the worlds fastest data-center GPUs (8799/5232 MLUPs/s on the Nvidia A100/V100 @ D3Q19 SRT FP32), on gaming GPUs and CPUs and it even runs my smartphone. For more performance details, see: https://arxiv.org/pdf/2112.08926.pdf #FluidX3D #OpenCL #Raytracing #GPU #LBM$

How aerodynamic is Boeing 767-400ER? (return from SC24, landing in Munich)

Formula 1 Ferrari SF71H in FluidX3D CFD at 10 Billion Cells with Wheel Rotation - 71TB CFD Data