I wrote this Real Time RayTracer from Scratch in 1 Week. Hybrid Rasterization + Raytracing Demo Dr. Moritz Lehmann 2026-09-19 | I wrote this Real Time RayTracer from Scratch in 1 Week. Hybrid Rasterization + Raytracing Demo
3 different GPUs, 1 CFD simulation - FluidX3D SLI-ing (Intel A770 + Intel B580 + Nvidia Titan Xp) Dr. Moritz Lehmann 2024-12-23 | My #FluidX3D #CFD software can "SLI" any GPUs together, regardless of microarchitecture or vendor, as long as VRAM capacity and bandwidth are similar. Here I'm running FluidX3D on 3 different GPUs:- Intel Arc A770 16GB (Alchemist)- Intel Arc B580 12GB (Battlemage)- Nvidia Titan Xp 12GB (Pascal)12GB + 12GB + 12GB VRAM are pooled together via domain decomposition, allowing for one large CFD simulation using 36GB combined VRAM, to fit 678 Million grid cells. This is made possible the most powerful #GPU programming language, #OpenCL.FluidX3D is available on #GitHub, for free: github.com/ProjectPhysX/FluidX3DThe model in this simulation is Santa's sleigh - with some X-wing modifications. Merry Christmas! :)The CAD model is from Zannyth / Kevin Piper: thingiverse.com/thing:2632246/files
What it looks like to cross the Atlantic Ocean on an airplane (Munich-Atlanta, work travel to SC24) Dr. Moritz Lehmann 2024-12-15 | ... and it goes on like this for ~9 hours, just water, water, and more water in all directions. The Atlantic Ocean is gigantic.
How aerodynamic is Boeing 767-400ER? (return from SC24, landing in Munich) Dr. Moritz Lehmann 2024-12-15 | 8x timelapse of landing in Munich after crossing the Atlantic Ocean over night, on the return from SC24 conference in Atlanta.
DLR RFZ Reusable Rocket Model in 4.3 Billion cell CFD - FluidX3D on 2x Intel Xeon 6979P Dr. Moritz Lehmann 2024-11-21 | #SC24 day 4! What does it look like when a reusable rocket booster slows down in Earth's atmosphere? I've simulated DLR's RFZ rocket model in #FluidX3D. 2x #Intel Xeon 6979P CPUs chew through this 4.3 Billion cell simulation in ~14 hours! 🔥 Simulation Stats:- FluidX3D CFD software: github.com/ProjectPhysX/FluidX3D- Intel CPU Runtime for OpenCL (oneAPI)- Lattice Boltzmann (LBM), D3Q19 SRT, FP32 arithmetic, FP32 memory storage- 1624×1624×1624 = 4.3 Billion grid cells, 1 cell = (2.6 mm)³- 380 GB memory footprint (55 Bytes/cell, or 19M cells per 1GB)- 22224 time steps = 0.05 seconds real time- 3x 1200 4k images rendered, velocity-colored Q-criterion isosurfaces visualized- 100 m/s airspeed- Reynolds number = 321M- Runtime = 13h53m(total) = 7h28m (LBM compute) + 6h24m (rendering)- Average LBM performance = 3541 MLUPs/sHardware Specs:- 2x Intel® Xeon® 6979P Prozessor (Granite Rapids), 2x 120 P-Cores, 2x 504MB Cache: ark.intel.com/content/www/us/en/ark/products/240781/intel-xeon-6979p-processor-504m-cache-2-10-ghz.html- 24x 128GB 8800MT/s MRDIMMs, for 3TB total RAM at 1.7TB/s- 0x #GPUsDLR RFZ rocket model: doi.org/10.5281/zenodo.11045747
Raindrop splash raytraced in 8K - FluidX3D CFD on 2x Intel Xeon 6979P Dr. Moritz Lehmann 2024-11-20 | #SC24 day 3! While any #GPU's (unified) VRAM pool today is still very limited, CPUs don't have this problem. In a dual-socket #Intel Xeon 6 system you can plug in up to 6TB of MRDIMMs at 1.7TB/s bandwidth, and it is all unified memory.This is particularly useful for algorithms that cannot be split up into multiple memory domains, such as the fast ray-grid-traversal Hashtag#raytracing in #FluidX3D. Here I did the raindrop splash on 2x Intel Xeon 6979P, at 4.3 Billion cells fitting in 380GB (without FP16 memory compression). The server simulated this in ~8 hours, and the raytracing in 8K resolution took ~19 hours.FluidX3D contains state-of-the-art volume-of-fluid and surface tension models for highly accurate free surface simulations. Combined with my own #OpenCL graphics engine, results are raytraced on-the-fly at image resolution as large as remaining memory can hold.FluidX3D on #GitHub: github.com/ProjectPhysX/FluidX3DThe simulation here (28094 time steps at 1716×1716×1458 resolution) took 8h16m, and 8K raytracing (4x 1200 frames) 18h39m. 8K means every frame is 33 MegaPixels. For every pixel, up to 10 rays are computed for 2 consecutive reflections/refractions. The entire simulation domain gets refracted through every droplet. The png images alone fill 48GB. Working with such large image/video resolution gets quite difficult, as no free video editing software supports it besides ffmpeg.This was computed on a single Intle Xeon 6 server, with 2x Intel Xeon 6979P 120-core CPUs with 3TB 8800MT/s MRDIMMs.Setup and parameters for reproducibility: github.com/ProjectPhysX/FluidX3D/blob/master/src/setup.cpp#L1088For validation and numerical details, see my dissertation: doi.org/10.15495/EPub_UBT_00006977Timestamps:0:00 view 10:20 view 20:40 view 31:00 view 4
Chevrolet C10 1961 in 4.3 Billion cell CFD - FluidX3D on 2x Intel Xeon 6979P Dr. Moritz Lehmann 2024-11-19 | #SC24 day 2! Here's a "smaller" #FluidX3D #CFD simulation, the Chevrolet C10 1961, at 4.3 Billion cells. Probably one of the most un-aerodynamic trucks out there, but that makes the airflow the more interesting! 2x Hashtag#Intel Xeon 6979P 120-core CPUs computed+rendered this in ~28 hours.Simulation Stats:- FluidX3D CFD software: github.com/ProjectPhysX/FluidX3D- Intel CPU Runtime for OpenCL (oneAPI)- Lattice Boltzmann (LBM), D3Q19 SRT, FP32 arithmetic, FP32 memory storage- 1389×2777×1111 = 4.3 Billion grid cells, 1 cell = (2.4 mm)³- 380 GB memory footprint (55 Bytes/cell, or 19M cells per 1GB)- 48750 time steps = 0.6 seconds real time- 1800 4k images rendered, velocity-colored Q-criterion isosurfaces visualized- 50 km/h airspeed- Reynolds number = 1.9M- Runtime = 1d03h58m (total) = 1d00h15m (LBM compute) + 3h43m (rendering)- Average LBM performance = 2261 MLUPs/sHardware Specs:- 2x Intel® Xeon® 6979P Prozessor (Granite Rapids), 2x 120 P-Cores, 2x 504MB Cache: ark.intel.com/content/www/us/en/ark/products/240781/intel-xeon-6979p-processor-504m-cache-2-10-ghz.html- 24x 128GB 8800MT/s MRDIMMs, for 3TB total RAM at 1.7TB/s- 0x #GPUs
Largest CFD sim ever on a single computer: NASA X-59 @ 117 Billion cells - FluidX3D on 2x Xeon 6980P Dr. Moritz Lehmann 2024-11-18 | This is the largest computational fluid dynamics (#CFD) simulation ever on a single computer, the #NASA X-59 jet at 117 Billion grid cells, fitting in 6TB RAM. This video visualizes 7.6 PetaByte (7.6 Million GB) of volumetric data.As a little gift to you all: #FluidX3D v3.0 is out now, enabling 31% larger grid resolution when running on CPUs or iGPUs, by fusing #OpenCL host+device buffers as zero-copy buffers. This optimization reduces memory footprint on CPUs/iGPUs from 72 to 55 Bytes/cell: github.com/ProjectPhysX/FluidX3D/releases/tag/v3.0#Intel #Xeon 6 with 8800MT/s MRDIMMs brings a new era of #HPC, where the memory capacity of a computer is measured in TeraByte, not GigaByte, with the massive 1.7TB/s memory bandwidth to back that up. Now such super large simulations are feasible on a single compact, energy-efficient CPU server, without having to change a single line of code thanks to #OpenCL. No #GPUs required! 🖖😋🟦Simulation Stats:- FluidX3D CFD software: github.com/ProjectPhysX/FluidX3D- Intel CPU Runtime for OpenCL (oneAPI)- Lattice Boltzmann (LBM), D3Q19 SRT, FP32 arithmetic, FP16C memory compression- 4062×12185×2369 = 117 Billion grid cells, 1 cell = (3.228 mm)³- 6.15 TB memory footprint (55 Bytes/cell, or 19M cells per 1GB)- 51627 time steps = 0.2 seconds real time- 5400 4k images rendered, velocity-colored Q-criterion isosurfaces visualized- 300 km/h airspeed, 10° angle of attack- Reynolds number = 51M, y+ = 538- Runtime = 30d23h23m (total) = 18d06h23m (LBM compute) + 12d16h59m (rendering)- Average LBM performance = 3836 MLUPs/sHardware Specs:- 2x Intel® Xeon® 6980P Prozessor (Granite Rapids), 2x 128 P-Cores, 2x 504MB Cache: ark.intel.com/content/www/us/en/ark/products/240777/intel-xeon-6980p-processor-504m-cache-2-00-ghz.html- 24x 256GB 8800MT/s MRDIMMs (Micron), for 6TB total RAM at 1.7TB/s bandwidth- 0x GPUsNASA X-59 model: nasa3d.arc.nasa.gov/detail/X-59
FluidX3D simulating 1.5 Billion cells for 1 Million time steps on a single Nvidia GH200 94GB GPU Dr. Moritz Lehmann 2024-08-30 | Simulating airflow is hard. Simulating water flow is harder. In addition to a Navier-Stokes solver, an accurate model for the free surface is needed, with another model for surface tension. Calculating the curvature of the water surface seems easy but is one of the hardest problems in a grid-based solver; I spent a big part of my Master's thesis on an efficient algorithm for this. And then gigantic amounts of brute force are required to get the surface and tiny water droplets resolved.This simulation of water flow through a 25cm radius baffled cylinder is 1.5 Billion cells, simulated for a massive 1 Million time steps. Such a large runlength for a Billion-cell simulation for the longest time was impossible, and still is unfeasible with even the most expensive commercial #CFD solvers. It is an unfathomable amount of number crunching.#FluidX3D does that on a single #GPU, in 2 days.Find FluidX3D on GitHub, it's free: github.com/ProjectPhysX/FluidX3DI performed this simulation on an Nvidia GH200 94GB GPU, using #OpenCL.
Lightning fast CFD free and accessible for everyone - 2 years of FluidX3D on GitHub Dr. Moritz Lehmann 2024-08-04 | With the superpowers of #OpenCL, I make lightning fast #CFD free and accessible for everyone. 🖖🧐👉 github.com/ProjectPhysX/FluidX3DTwo years ago today I published #FluidX3D on #GitHub, not knowing where the journey would go. Since then I've made it cross-vendor multi-GPU capable - on any hardware since 2009. I've doubled down on compatibility - full interactive in-situ rendering support on #Linux and Windows, CLI support on Android and macOS. I've sped up compiling and start-up to just seconds. I've added fast voxelization, particles, improved real-time visualization, and extensive documentation. I've eliminated every single bug, delivering on my zero-bug policy. I've future-proofed the software for all of tomorrow's massive hardware with multi-TeraByte memory capacity.Today FluidX3D is the #1 CFD software on GitHub, #6 in #HPC & #GPGPU, #9 in OpenCL, #4 in #raytracing. Thank you for using FluidX3D, let's keep going! 🖖😋Timestamps:0:00 Interstellar Ranger0:09 liquid metal on speaker0:15 X-Wing0:18 Tie-Fighter0:21 Space Shuttle0:25 City0:28 Ferrari SF71H0:38 Electric Ducted Fan0:54 Hydraulic Jump1:01 Bell 222 Airwolf1:16 Ghostbusters Ecto-11:24 Quadcopter1:44 Arm Chair1:54 Concorde2:04 Delta Wing2:28 Sphere2:40 Raindrop3:00 Outro
How realistic can a simulation be? - FluidX3D CFD raytraced in 8K Dr. Moritz Lehmann 2024-03-24 | FluidX3D on #GitHub: github.com/ProjectPhysX/FluidX3DHow realistic can a #CFD simulation be? Here is a 1 billion cell #FluidX3D simulation of an impacting raindrop, fully raytraced in 8K. FluidX3D contains state-of-the-art volume-of-fluid and surface tension models for highly accurate free surface simulations. Combined with my own #OpenCL graphics engine, results are raytraced on-the-fly at image resolution as large as remaining #GPU VRAM can hold.The simulation here (17355 time steps at 1060x1060x900 resolution) took only 1 hour, but 8K raytracing (4x 1200 frames) almost 5 hours. 8K means every frame is 33 MegaPixels. For every pixel, up to 10 rays are computed for 2 consecutive reflections/refractions. The entire simulation domain gets refracted through every droplet. The png images alone fill 48GB. Working with such large image/video resolution gets quite difficult, as no free video editing software supports it besides ffmpeg.This was computed on a single #AMDInstinct MI210 64GB #HPC GPU, which has neither dedicated #raytracing hardware nor any sort of #rendering hardware. FluidX3D can still render/raytrace using OpenCL.Setup and parameters for reproducibility: github.com/ProjectPhysX/FluidX3D/blob/master/src/setup.cpp#L1088For validation and numerical details, see my dissertation: doi.org/10.15495/EPub_UBT_00006977Timestamps:0:00 view 10:20 view 20:40 view 31:00 view 41:20 view 1 zommed in1:40 view 2 zommed in2:00 view 3 zommed in2:20 view 4 zommed in
FluidX3D v2.14 Update Trailer Dr. Moritz Lehmann 2024-03-03 | #FluidX3D has been a visualization powerhouse from the beginning, and with update v2.14 it got even better: Now velocity, density and temperature fields can be visualized in field, slice, streamline, and Q-criterion isosurface modes, in real time while the #CFD simulation is running. All of this #OpenCL rendering works on multi-#GPU systems too, with any combination of #Nvidia/#AMD/#Intel GPUs.Full release notes: github.com/ProjectPhysX/FluidX3D/releases/tag/v2.14The video is some screen recordings to showcase the update, all real-time simulations on #IntelArc A750:0:00 Delta Wing0:58 Cow Aerodynamics1:43 2D Karman Vortex Street1:49 Turbulence around Sphere at Re=1M2:02 Raytraced Water Sloshing2:38 Cessna 1723:02 3D Rayleigh-Benard Convection4:32 OutroTo give you some insight in development: This update was ~650 lines of code, ~5% of the FluidX3D code base. Implementation is the easiest and quickest part; I spend most time on testing and validation, because I do not tolerate bugs in my software. Just as I had checked that all new rendering kernels were working on my Arc A750, I tested again on a GTX 960M and realized it wasn't working there - because of another bug in Nvidia's drivers (vector types like int3 as OpenCL kernel parameters are broken). This was the 7th bug I have discovered in Nvidia's OpenCL drivers so far (and reported to them). A workaround later all was good on the Nvidia GPU too, and I could finally publish the update. Maintaining compatibility across all hardware takes large effort, but it's worth it, as it gives users the freedom to choose any hardware without #CUDA/HIP vendor lock-in.
FluidX3D can SLI together 🔵 Intel Arc A770 + 🟢 Nvidia Titan Xp - through the power of OpenCL Dr. Moritz Lehmann 2024-02-02 | This is wild: FluidX3D can "SLI" together an 🔵 Intel Arc A770 + 🟢 Nvidia Titan Xp, pooling 12GB+12GB of their VRAM for one large 450 million cell CFD simulation. Top half computed+rendered on A770, bottom half computed+rendered on Titan Xp. They seamlessly communicate over PCIe. Performance is ~1.7x of what either GPU could do on its own.OpenCL shows its true power here - a single implementation works on literally all GPUs at full performance, even at the same time. I have specifically designed FluidX3D for cross-vendor multi-GPU, to allow combining any GPUs as long as VRAM capacity and bandwidth are similar.Now that I have some new hardware, I can finally demonstrate this in practice. This setup is turbulence created by a sphere at Re=1M. 532×1600×532 resolution in 2×12GB VRAM, 64k time steps, 1.5h for compute+rendering.Find the FluidX3D CFD software on GitHub: github.com/ProjectPhysX/FluidX3DHow does cross-vendor multi-GPU work?Each GPU computes only half of the simulation box. VRAM capacity and bandwidth are similar (A770: 16GB@560GB/s, Titan Xp: 12GB@548GB/s) such that the compute time for both domains is similar. Where the two GPU domains (each 8.6 GB in size) touch, some data has to be exchanged. These layers (8.5 MB in size) are first extracted within each GPU's VRAM into transfer buffers. The transfer buffers are copied from VRAM to CPU RAM over PCIe (A770: PCIe 4.0 x8 (~8GB/s), Titan Xp: PCIe 3.0 x8 (~4GB/s). The CPU waits for all transfer buffers to arrive, and then only swaps their pointers. Afterwards, transfer buffers are copied back over PCIe to the GPUs, are inserted back into the domains within VRAM, and each GPU can again compute LBM on its own domain. Because OpenCL only needs the generic PCIe interface and not proprietary SLI/Crossfire/NVLink/InfinityFabric, this works with any combination of Intel/Nvidia/AMD GPUs.Here is the simplified multi-GPU communication architecture illustrated:.-----------------------------------------------------------------.| GPU 0 || LBM Domain 0 |'-----------------------------------------------------------------' | selective /|\ \|/ in-VRAM copy | .-------------------------------------------------------. | GPU 0 - Transfer Buffer 0 | '-------------------------------------------------------' | PCIe /|\ \|/ copy | .-------------------------. .-------------------------. | CPU - TB 0 | | CPU - TB 1 | '-------------------------'\ /'-------------------------' pointer X swap .-------------------------./ \.-------------------------. | CPU - TB 1 | | CPU - TB 0 | '--------------------------' '--------------------------' /|\ PCIe | | copy \|/ .-------------------------------------------------------. | GPU 1 - Transfer Buffer 1 | '-------------------------------------------------------' /|\ selective | | in-VRAM copy \|/.-----------------------------------------------------------------.| GPU 1 || LBM Domain 1 |'-----------------------------------------------------------------' | domain synchronization barrier |-----------------------------------------------------→ timeTimestamps0:00 intro0:04 simulation#FluidX3D #CFD #OpenCL #Intel #IntelArc #Nvidia #FluidSimulation #turbulence
Raytraced fluid simulation on Intel Arc A750 GPU Dr. Moritz Lehmann 2024-01-19 | I haven't posted a simulation in a while. It's #SimulationFriday again, so here we go again! After many updates and an overhaul to my OpenCL graphics engine, this is the raindrop splash simulated and raytraced on the Arc A750 at 109 million cells (504x504x428). Took 16 minutes for 8252 time steps (3ms real time).Find the FluidX3D CFD software on GitHub: github.com/ProjectPhysX/FluidX3DThis particular setup is here: github.com/ProjectPhysX/FluidX3D/blob/master/src/setup.cpp#L1071Timestamps0:00 Camera 10:05 Camera 20:10 Camera 30:15 Camera 4#FluidX3D #CFD #IntelArc #Raytracing #FluidSimulation
Ghostbusters Ecto-1 is haunted by Aerodynamic Drag Dr. Moritz Lehmann 2023-08-18 | - FluidX3D CFD software: github.com/ProjectPhysX/FluidX3D- 1070×3060×918 = 3 Billion cells resolution- 60km/h speed- 1 second real time- 4.5 hours runtime on 4x Nvidia A100 SXM 40GB (simulation + rendering)- CAD model: thingiverse.com/thing:5232650/filesTimestamps0:00 rotating camera1:00 follow view2:00 front view#FluidX3D #Ghostbusters #CFD #GPU #Nvidia #A100
FluidX3D on GigaIO SuperNODE - Concorde 40 Billion Cell CFD Simulation - 33h on 32x MI210 64GB GPUs Dr. Moritz Lehmann 2023-08-01 | This is one of the largest CFD simulations ever done, on the world's largest GPU server, the GigaIO SuperNODE, equipped with 32x AMD Instinct MI210 64GB GPUs, for a total 2TB VRAM.gigaio.com/supernodeThe simulation shows the 62m long Concorde before landing at 300km/h airspeed and 10° angle of attack, for 1 second in flight. The Reynolds number based on wingspan is 146 Million.The simulation resolution is 2976×8936×1489 = 40 Billion cells, with a tiny cell size of (12.4mm)³. 67268 time steps were computed in 29 hours, plus 4 hours for rendering 5×600 4K frames, for a total runtime of 33 hours. The video shows velocity-magnitude colored Q-criterion isosurfaces. A single frame of the velocity field is 475GB, so the 600 frames visualize a total of 285TB data.This is a test of the newly implemented free-slip boundaries, which are a more accurate model for the turbulent boundary layer than no-slip boundaries.On the same hardware, commercial CFD software like Ansys or Star-CCM+ would need several years of compute time for such a simulation. FluidX3D does it over the weekend.The FluidX3D source code is on GitHub, and the software is free for non-commercial use: github.com/ProjectPhysX/FluidX3DConcorde model: thingiverse.com/thing:1176931/filesTimestamps0:00 front view0:10 follow view0:20 wing view0:30 top view0:40 side view#FluidX3D #Concorde #CFD #GPU #AMDInstinct
Airodynamics of an arm chair on 4x A100 GPUs Dr. Moritz Lehmann 2023-07-28 | I had 2 days and ~8000 core-hours left on the supercomputer and didn't know what to do with my life. So I decided to push the frontiers of science with the most relevant computational fluid dynamics simulation of all time.This is a simulation of an arm chair at 50km/h wind speed, for 1 second real-time. 3 billion grid cells resolution (1262 x 2522 x 946), 4x Nvidia A100 40GB SXM4 GPUs, 65694 time steps, 3 hours 11 minutes runtime for both compute and rendering.The simulation software used for this is FluidX3D: github.com/ProjectPhysX/FluidX3D
Most detailed Quadcopter CFD Simulation ever at 3 Billion Cells - FluidX3D - 8 Hours on 4x A100 40GB Dr. Moritz Lehmann 2023-07-07 | Here is the most detailed simulation of a #quadcopter ever, at 3 billion microscopic (0.2mm)³ grid cells. With #FluidX3D we can finally have such high-quality multirotor #CFD simulations. Compute time was 8 hours on 4 Nvidia #A100 40GB GPUs, for 86k timesteps with rotor revoxelization (46k triangles each) every 4 timesteps, and 3 frames rendered every 48 timesteps. Bottom line, 9 billion cells updated per second despite rotor revoxelization and rendering.At 8000 RPM, 12cm blade diameter and 1cm chord length, the Reynolds number here is fairly moderate at 34k, which allows for DNS-LES simulation.The FluidX3D source code is on GitHub: github.com/ProjectPhysX/FluidX3DHappy #SimulationFriday!Timestamps:0:00 side0:30 front0:60 bottomKudos to @MusaWilmar for the awesome CAD model: thingiverse.com/thing:2540161/filesThanks to Jülich Supercomputing Centre JURECA-DC Hardware Evaluation Platform for providing me access to the A100 GPU node!
OpenCL GPU Programming for HPC Applications (ChEESE Webinar - Center of Excellence in Solid Earth) Dr. Moritz Lehmann 2023-07-03 | My Webinar talk at ChEESE - Center of Excellence in Solid Earth on 28.06.2023.Start coding OpenCL here: github.com/ProjectPhysX/OpenCL-WrapperTimestamps0:00:00 Intro by Giorgio0:02:17 Start0:02:47 Who am I?0:05:41 My OpenCL Project0:07:14 OpenCL History 0:08:29 Big Advantages of OpenCL0:10:40 CPU vs. GPU0:13:45 OpenCL Memory Model0:15:41 OpenCL Platform/Device/Context/Program0:18:59 OpenCL Code Structure0:20:53 OpenCL C Language0:23:02 A simple OpenCL C Kernel0:24:46 Functions in OpenCL C0:25:26 OpenCL C++ API0:26:55 Coalesced/Misaligned Memory Access0:29:49 Branching in OpenCL - bad?0:32:01 The Race Condition - a GPU-specific Bug0:34:35 Roofline Model0:45:08 OpenCL Workgroups0:46:29 Register Pressure0:49:52 Cache Tiling - local Memory Optimization0:51:11 Multi-GPU with OpenCL0:58:14 Case Study: Solid Voxelization of Triangle Mesh1:05:29 Sime final Tipps1:12:20 Question: GPU-aware MPI?1:15:29 Question: OpenCL vs. CUDA Performance?1:16:35 Question: How portable is OpenCL REALLY?1:19:12 Question: Are all FLOPs created equal?1:22:17 Question: How to use local Memory?1:26:11 Question: How to profile OpenCL?1:29:05 Question: Race Condition - Mutex?1:32:13 Conclusions#OpenCL #GPU #FluidX3D
How I earned my Physics PhD at age 25, with summa cum laude Dr. Moritz Lehmann 2023-06-29 | I had a recording from one of the practice runs of my #PhD defense talk, and I've finally uploaded it to YouTube! Enjoy!My PhD thesis is published here: doi.org/10.15495/EPub_UBT_00006977Timestamps:00:00 Intro00:23 Start00:42 Part A: Microplastic Studies02:21 A1: Raindrop Study10:23 A2: Bubble Study12:35 Part B: LBM Memory Optimizations14:04 B1: Esoteric-Pull Streaming15:56 B2: FP16 Memory Compression17:25 B3: Combined Simulation+Rendering18:26 Conclusions#PhD #Physics #FluidX3D #CFD
There is a horrible bug in my code. Dr. Moritz Lehmann 2023-06-25 | This is the second time this happened to me.
Bell 222 Helicopter in FluidX3D CFD - 10 Billion Cells, 71TB vizualized - 6.4 hours on 8x MI200 GPUs Dr. Moritz Lehmann 2023-06-23 | I've always wanted to do a helicopter! 🖖😎🚁Here it is, a Bell 222 in #CFD. With a massive 10 billion cells resolution, 71 TeraByte data visualized. Just for fun, because I can! Enjoy this #SimulationFriday gem! 🖖😋I ran this on 8x AMD Instinct #MI200 #GPUs. 3016×3616×905 cells, 75k time steps. Runtime was a mere 6 hours and 23 minutes for both simulation and rendering. In real time, this is 0.34 seconds, or 1.5 revolutions of the main rotor at 348 RPM.You can try this software too, it's called #FluidX3D, and it's on #GitHub: github.com/ProjectPhysX/FluidX3DTimestamps:0:00 side0:10 rotor0:20 front0:30 back0:40 top0:50 followThe Bell 222 CAD model is from @thingiverse and was edited with Microsoft 3D Builder: thingiverse.com/thing:1625155/filesThanks to Jülich Supercomputing Centre JURECA-DC Hardware Evaluation Platform for providing me access to the MI200 GPU node!
Cessna 172 in FluidX3D CFD - 220 million cells - 10 minutes on Titan Xp GPU Dr. Moritz Lehmann 2023-06-19 | #FluidX3D source code: github.com/ProjectPhysX/FluidX3DLet's leave real time and bump up resolution for the Cessna 172 #CFD, from 1GB to 12GB! 220 million cells, 5600 time steps, 0.1s flight at 226km/h cruising speed (Re=46M). 10 minutes compute+render time on the old Titan Xp #GPU, the 1080 Ti's bigger brother.Real time simulation video: youtu.be/Gd_MEHPecw0Although this is 10.9x the VRAM occupation, lateral resolution increased only by 2.2x. For every doubling in lateral resolution, VRAM footprint increases by 8x and compute time increases by 16x. This is what makes high-resolution CFD so challenging.Source for the CAD model: thingiverse.com/thing:814319/files
Real time CFD with FluidX3D - Cessna 172 - 20 million cells - Titan Xp GPU Dr. Moritz Lehmann 2023-06-16 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DIt's #SimulationFriday again! Here is a real time #FluidX3D #CFD simulation of a Cessna 172, 20 million cells, on my old Titan Xp Pascal #GPU. 33 lines of code for this setup.This is a preview for the upcoming #FluidX3D v2.8 update. I'm improving handling of composite geometries with moving parts, and I'm adding semi-transparent rendering.Source for the CAD model: thingiverse.com/thing:814319/files
FluidX3D v2.7 Update: New Slice and Surface Pressure Data Visualization Modes Dr. Moritz Lehmann 2023-05-29 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DIn the #FluidX3D v2.7 update I added slice and surface pressure visualization modes. This was 1 day of programming for me!The video shows a 25 million cell #CFD simulation of the Concorde in real time with interactive visualization, running on an Nvidia Titan Xp Pascal #GPU from 2017.
FluidX3D - A New Era of Computational Fluid Dynamics Dr. Moritz Lehmann 2023-05-12 | With slow commercial #CFD software, compute time for my PhD studies would have exceeded decades. The only way to success was to master #GPU programming, write my own CFD solver from the ground up and turn runtime to days. Now a 10 billion cell simulation (like in the video) runs over night on an 8-GPU server, visualizing hundreds of TeraByte with my #OpenCL graphics engine.I have since published my source code to make this powerful tool freely available to other researchers/students/hobbyists. Today #FluidX3D the most popular CFD code on GitHub with 1.6k Stargazers: github.com/ProjectPhysX/FluidX3DThanks to the people at Jülich Supercomputing Centre for letting me test their hardware!Music: Odyssey - JIGLR: youtu.be/oi1hxQLZbco
Running FluidX3D on Intel Arc A750 with interactive ASCII graphics over remote SSH Dr. Moritz Lehmann 2023-04-24 | Here FluidX3D runs on an Intel Arc A750 GPU on a remote Linux PC. The simulation (raindrop impact, 125 million cells) runs in real-time and is visualized interactively. Graphics are donwscaled, converted into colored ASCII and passed through SSH into the WSL terminal. Piece of cake!The entire source code of FluidX3D is public on GitHub: github.com/ProjectPhysX/FluidX3D#FluidX3D #CFD #IntelArc
Real-Time Fluid Raytracing with OpenCL (on an old GTX GPU) Dr. Moritz Lehmann 2023-04-11 | This is a demo of a hydraulic jump simulation with FluidX3D, running on a single Nvidia Titan Xp GPU, which lacks RTX. The grid resolution is 96x352x96 (3.2 Million) cells. Both simulation and raytracing run in real time.The simulation software used is FluidX3D, the fastest and most memory efficient lattice Boltzmann CFD software, running on all GPUs via OpenCL. Find FluidX3D on GitHub: github.com/ProjectPhysX/FluidX3DThe latest update FluidX3D v2.5 featured in this demo adds light attenuation in the fluid and improved raytracing performance when the camera is inside the fluid. Full changelog: github.com/ProjectPhysX/FluidX3D/releases/tag/v2.5#FluidX3D #OpenCL #Raytracing #GPU #LBM
Macro timelapse of water vapor deposition to ice crystals (16x) and melting (2x) Dr. Moritz Lehmann 2023-03-25 | 0:00 vapor deposition0:10 meltingThe metal bar is 0.5mm wide. The rapid cooldown is achieved with 2 Peltier/TEC elements stacked on a CPU cooler.
12 hours to 147 seconds - the FluidX3D speedup to competition Dr. Moritz Lehmann 2023-02-09 | Just a low resolution test to match what competition is doing, 55M cells & 16600 time steps. Simcenter takes 12 hours for this on 8x Nvidia A100 [1], FluidX3D takes 147 seconds on only one A100. That's 2350x faster. Download FluidX3D for free: github.com/ProjectPhysX/FluidX3D[1] blogs.sw.siemens.com/simcenter/cfd-speedup-with-simplecPorsche Panamera model: thingiverse.com/thing:2039291/files
Pushing the limits of whats possible in CFD - 10 billion cells, 214TB visualized, 14h on 1 computer Dr. Moritz Lehmann 2023-01-26 | Find FluidX3D on GitHub: github.com/ProjectPhysX/FluidX3DThis is a lattice Boltzmann simulation of a 10cm diameter electric ducted fan (EDF) at 28000 RPM, which is Re=1M at the blade tips. 10 billion grid cells (1876x2814x1876) are simulated for 180k time steps. Revoxelization of the fan is done every 10 time steps, frame rendering every 100 time steps. Rendered is the velocity field (12 Byte/cell) on 1800 frames, which comes in at 214TB total visualized data. YouTube 4K video compression can hardly handle so much detail.Total compute time (DNS-LES simulation with LBM, re-voxelization and rendering combined) was 14 hours on a single GPU server with 8x AMD Instinct MI200, at a total 512GB VRAM. Energy consumption was about 40kWh. To reproduce this simulation, here is the source code for this setup: github.com/ProjectPhysX/FluidX3D/blob/master/src/setup.cpp#L440Source of the EDF geometry: thingiverse.com/thing:3014759/files
Radial Fan simulated in FluidX3D CFD (this was 45min on a gaming PC) Dr. Moritz Lehmann 2023-01-19 | Find FluidX3D on GitHub: github.com/ProjectPhysX/FluidX3D1200 rendered frames, 4 revoxelizations per frame, 10 time steps per revoxelization. 220 million cells, 3.2TB visualized data. An average gaming #GPU (Titan Xp 12GB) does this #FluidX3D #CFD #simulation in 45 minutes. For $0. Any other CFD software takes weeks.Radial fan .stl file source: thingiverse.com/thing:6113/files
The raindrop again but with 8 billion (!) cells in 512GB GPU memory Dr. Moritz Lehmann 2023-01-06 | I have now sucessfully extended FluidX3D to multi-GPU, allowing to pool VRAM of multiple GPUs installed in a server node via domain decomposition, here 8x AMD Instinct MI200 64GB. The multi-GPU update is now published on GitHub: github.com/ProjectPhysX/FluidX3DThis is an 8 billion grid cell simulation of a d = 7 mm diameter terminal velocity (9.55 m/s) raindrop impact at 20° inclination, simulated for 3 milliseconds (34650 LBM time steps). The box is 5d x 5d x 4.25d and the pool height is 2d. For reproducibility: The dimensionless numbers for this setup are Re = 63618, We = 8864, Fr = 36.44, Ca = 0.1393, Bo = 6.674.Velocity, density and fluid mass are solved at 8 billion lattice points (2122 x 2122 x 1804) with D3Q19 SRT and FP16S memory compression. The simulation requires 512 GB video memory and was conducted on 8x AMD Instinct MI200 GPUs at Jülich Supercomputing Centre (JSC) JURECA-DC-MI200. Combined compute+render time was 330 minutes. 4K60 rendering is done via domain decomposition rasterization, where each GPU renders only its own domain and the frames are seamlessly overlayed via their z-buffers. 4x 451 frames (6.3GB) are generated. Find physical accuracy validation of the raindrop setup in this paper: doi.org/10.1186/s43591-021-00018-8How is it possible to squeeze 8 billion grid points in only 512GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 67 Bytes/node for D3Q19 LBM with the Volume-of-Fluid extension, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data ever has to be copied to the CPU or hard drive, but only rendered frames instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 front view0:07 side view0:15 back view0:22 top view#CFD #GPU #FluidX3D #OpenCL
Formula 1 Ferrari SF71H in FluidX3D CFD at 10 Billion Cells with Wheel Rotation - 71TB CFD Data Dr. Moritz Lehmann 2022-11-03 | The wheels are now rotating in this simulation, via moving bounce-back boundaries. Settling a debate, started by my other video of the F1 car where wheels were not rotating: youtu.be/q4rNIbqvyQIHow much does wheel rotation affect airflow? When the wheels rotate, the turbulent boundary layer detaches a bit earlier from the tire surface. This creates significantly more turbulence.FluidX3D source code: github.com/ProjectPhysX/FluidX3DThe 10 seconds of video show the car driving at 100km/h for 0.5 seconds. Moving boundaries impose no additional computational cost. This took 7 hours to compute+render on 8x AMD Instinct MI200 GPUs. Grid resolution is 2144×4288×1072, and 108360 time steps are computed, with 3x 4K images rendered every 180 LBM time steps. Shown is velocity-colored Q-criterion isosurfaces with marching-cubes. Reynolds number is 3.75 Million with Smagorinsky-Lilly subgrid model.How is it possible to squeeze 10 billion grid points in only 512GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. 1 frame of the velocity field is 118GB, 603 frames are generated, which would be 71TB. No volumetric data ever has to be copied to the CPU or hard drive, but only rendered 4K frames (33MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 side view0:10 front view0:20 top viewThanks to the people at Jülich Supercomputing Centre for letting me test their hardware!The 3D model is from Thingiverse: thingiverse.com/thing:2990512/files#CFD #GPU #FluidX3D #OpenCL
CFD timelapse of wind over city Dr. Moritz Lehmann 2022-10-14 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DTimelapse: 1km/h wind blowing over a city for about an hour, simulated with FluidX3D on a 3 billion voxel grid. OpenCL compute (1448×2896×724 resolution grid, 34k time steps) plus rendering took ~104 minutes. Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 19 Million with Smagorinsky-Lilly subgrid model.How is it possible to squeeze 3 billion grid points in only 160GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 35GB!) ever has to be copied to the CPU or hard drive, but only rendered 4K frames (33MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 bottom view0:20 top view0:40 side viewThanks to the people at Jülich Supercomputing Centre for letting me test their hardware!The 3D model of the city was provided by Thomas Tian: https://www.building-engineering.de/#CFD #GPU #FluidX3D #OpenCL
Visualizing 144 TeraByte of CFD data for fun Dr. Moritz Lehmann 2022-10-07 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DThe 20 seconds of video show the car driving at 100km/h for 1 second.Mattia Binotto's Ferrari SF71H in #CFD. In this 10 billion voxel #FluidX3D simulation you see the wild aerodynamic optimization for a very successful F1 car. #OpenCL compute (2152×4304×1076 resolution grid, 217k time steps) plus rendering 3x 20s 4K60 video took 14 hours. Shown is velocity-colored Q-criterion isosurfaces with marching-cubes. Reynolds number is 3.75 Million with Smagorinsky-Lilly subgrid model.How is it possible to squeeze 10 billion grid points in only 512GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. 1 frame of the velocity field is 120GB, 1201 frames are generated, which would be 144TB. No volumetric data ever has to be copied to the CPU or hard drive, but only rendered 4K frames (33MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 side view0:20 front view0:40 top viewThanks to the people at Jülich Supercomputing Centre for letting me test their hardware!The 3D model is from Thingiverse: thingiverse.com/thing:2990512/files#CFD #GPU #FluidX3D #OpenCL
1 Billion Cell Raindrop Impact CFD Simulation Dr. Moritz Lehmann 2022-09-29 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DFull video in 4K: youtu.be/VadLwt9OqMoTechnical talk on how I did it: youtu.be/pD8JWAZ2f8oThis is a 1 billion grid cell simulation of a d = 7 mm diameter terminal velocity (9.55 m/s) raindrop impact at 20° inclination, simulated for 3 milliseconds (17355 LBM time steps). The box is 5d x 5d x 4.25d and the pool height is 2d. For reproducibility: The dimensionless numbers for this setup are Re = 63618, We = 8864, Fr = 36.44, Ca = 0.1393, Bo = 6.674.Velocity, density and fluid mass are solved at 1 billion lattice points (1060 x 1060 x 900) with D3Q19 SRT and FP16S memory compression. The simulation requires 64 GB video memory and was conducted on an AMD Instinct MI200 GPU at Jülich Supercomputing Centre (JSC) JURECA-DC-MI200. Compute time alone was 61 minutes, plus 64 minutes rendering time (raytracing at 4K60 resolution with 1-10 rays per pixel; 4x 446 frames (11.8GB) generated). Find physical accuracy validation of the raindrop setup in this paper: doi.org/10.1186/s43591-021-00018-8How is it possible to squeeze 1 billion grid points in only 64GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 67 Bytes/node for D3Q19 LBM with the Volume-of-Fluid extension, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data ever has to be copied to the CPU or hard drive, but only rendered frames instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCL#shorts
1 Billion Cell Raindrop Impact CFD Simulation, 2 hours on 1 AMD Instinct MI200 64GB GPU Dr. Moritz Lehmann 2022-09-29 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DThis is a 1 billion grid cell simulation of a d = 7 mm diameter terminal velocity (9.55 m/s) raindrop impact at 20° inclination, simulated for 3 milliseconds (17355 LBM time steps). The box is 5d x 5d x 4.25d and the pool height is 2d. For reproducibility: The dimensionless numbers for this setup are Re = 63618, We = 8864, Fr = 36.44, Ca = 0.1393, Bo = 6.674.Velocity, density and fluid mass are solved at 1 billion lattice points (1060 x 1060 x 900) with D3Q19 SRT and FP16S memory compression. The simulation requires 64 GB video memory and was conducted on an AMD Instinct MI200 GPU at Jülich Supercomputing Centre (JSC) JURECA-DC-MI200. Compute time alone was 61 minutes, plus 64 minutes rendering time (raytracing at 4K60 resolution with 1-10 rays per pixel; 4x 446 frames (11.8GB) generated). Find physical accuracy validation of the raindrop setup in this paper: doi.org/10.1186/s43591-021-00018-8How is it possible to squeeze 1 billion grid points in only 64GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 67 Bytes/node for D3Q19 LBM with the Volume-of-Fluid extension, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data ever has to be copied to the CPU or hard drive, but only rendered frames instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 front view0:07 side view0:15 back view0:22 top view#CFD #GPU #FluidX3D #OpenCL
SpaceX Starship Belly Flop CFD Simulation in 160GB GPU VRAM Dr. Moritz Lehmann 2022-09-23 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DStarship doing the belly flop in #CFD. In this 3 billion voxel #FluidX3D simulation you see how turbulence+drag is maximized on purpose. #OpenCL compute (912×1824×1824 resolution grid, 108k time steps) took ~182 minutes, rendering 3x 20s 4K60 video took ~68 minutes. Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 1 Million with Smagorinsky-Lilly subgrid model.How is it possible to squeeze 3 billion grid points in only 160GB VRAM?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 35GB!) ever has to be copied to the CPU or hard drive, but only rendered 4K frames (33MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 bottom view0:20 top view0:40 side viewThanks to the people at Jülich Supercomputing Centre for letting me test their hardware!The 3D model is from Thingiverse: thingiverse.com/thing:4912729/files#CFD #GPU #FluidX3D #OpenCL
Space Shuttle 10 billion voxel CFD on 8x 64GB GPUs Dr. Moritz Lehmann 2022-09-02 | FluidX3D source code: github.com/ProjectPhysX/FluidX3DNow I get why the Space Shuttle sometimes was also called a "flying brick". This is a 10 billion voxel lattice Boltzmann CFD simulation on 4x AMD Instinct MI250 (8x MI200 GCD with 64GB VRAM each).Simulating 108k time steps on the 1608×4824×1280 resolution grid took 6 hours, plus 36 minutes for rendering 2x 30s 4K60 video. Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 1 Million with Smagorinsky-Lilly subgrid model.Grid resolution here is about 60x bigger than the largest Space Shuttle CFD simulation ever done by NASA.How is it possible to squeeze 10 billion grid points in only 512GB?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 114GB!) ever has to be copied to the CPU or hard drive, but only rendered 4K frames (33MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 bottom view0:30 top viewThanks to the people at Jülich Supercomputing Centre for letting me test their hardware!The 3D model is from Thingiverse: thingiverse.com/thing:4975964/files#CFD #GPU #FluidX3D #OpenCL
Interstellar Ranger 10 billion voxel CFD on 8x 64GB GPUs Dr. Moritz Lehmann 2022-08-30 | There is no time for caution when Cooper lands Ranger 1 on Miller's planet in Interstellar.FluidX3D source code: github.com/ProjectPhysX/FluidX3DThis is the largest CFD simulation ever done on a single GPU node, with 10 billion LBM grid points on 4x AMD Instinct MI250 (8x MI200 GCD with 64GB VRAM each).Simulating 36k time steps took 123 minutes at 2152×4784×968 grid resolution, plus 203 minutes for rendering 5x 20s 1080p video. Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 1 Million with Smagorinsky-Lilly subgrid model.How is it possible to squeeze 10 billion grid points in only 512GB?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 14GB!) ever has to be copied to the CPU or hard drive, but only rendered 1080p frames (8MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 side view0:20 follow view0:40 wing view1:00 window view1:20 front view#CFD #GPU #FluidX3D #OpenCL
Aerodynamics of a cow Dr. Moritz Lehmann 2022-08-19 | This video may not change your life, but my #FluidX3D software will, if you do research in #CFD. On the same #GPU(s) it's 100-2000x faster than expensive commercial solvers. The entire source code is on GitHub: github.com/ProjectPhysX/FluidX3DThis 10s video shows 10s in real time with 1m/s wind speed. 476×952×476 #LBM grid (215 million voxels), 28k time steps, 23 minutes for compute+rendering on my PC with Titan Xp GPU.How is it possible to squeeze 215 million grid points in only 12GB?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/6 of conventional LBM codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with #OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 2.5GB!) ever has to be copied to the CPU or hard drive, but only rendered 1080p frames (8MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCL
USS Enterprise CFD simulation on my gaming PC Dr. Moritz Lehmann 2022-08-12 | Live long and prosper! 🖖😎This is a quick 220 million voxel FluidX3D CFD simulation of the Star Trek USS Enterprise NCC-1701-E. OpenCL simulation+rendering took 15 minutes on my 2017 gaming PC with 12GB Titan Xp GPU.Grid resolution is 528×1584×264 and 18000 time steps are computed with a video frame every 30 steps. Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 100k with Smagorinsky-Lilly subgrid model.FluidX3D source code on GitHub: github.com/ProjectPhysX/FluidX3DHow is it possible to squeeze 220 million grid points in only 12GB?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 2.5GB!) ever has to be copied to the CPU or hard drive, but only rendered 1080p frames (8MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCL#CFD #GPU #FluidX3D #OpenCL
TIE fighter tumbling out of control - 1.24 billion voxel CFD simulation (AMD MI250) Dr. Moritz Lehmann 2022-08-04 | No wonder the Jedi are winning! The TIE fighter is tumbling out of control in atmospheric flight due to troublesome turbulence!1.24 billion voxel FluidX3D CFD simulation on 1 GCD (64GB) of an MI250*, showing off OpenCL performance on AMD Instinct.This took 6 hours for combined compute (50k LBM time steps at 852×1704×852 resolution, mesh re-voxelization every 28 time steps) and rendering (4x 30s 1080p60 video).Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 100k with Smagorinsky subgrid model, but at this resolution it would probably also run stable without.FluidX3D source code: github.com/ProjectPhysX/FluidX3DWhile the one GCD of MI250 simulates the X-wing, the second GCD can be used for the dark side of the force and simulate TIE fighter aerodynamics. Both of these large-scale simulations can run at the same time with single-socket AMD Instinct hardware. What a time to be alive!Star Wars X-wing simulation: youtu.be/3JNVBQyetMA*The MI250 is actually 2 entirely separate GPUs (GCDs) in a single socket with 64GB memory each. One GCD can't directly access the memory of the other. This simulation is only running on one GCD, using the full 64GB of its unified memory.To use both GCDs, or better all 8 GCDs in the server at Jülich Supercomputing Center, the code would need to be specifically extended for multi-GPU support. This is very difficult and time-consuming, and for some parts of FluidX3D, like raytracing graphics, it is close to impossible. Maybe one day I will find the time to do it, but not now.Still, how is it possible to squeeze 1.24 billion grid points in only 64GB?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 14GB!) ever has to be copied to the CPU or hard drive, but only rendered 1080p frames (8MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 head-on view0:30 top view0:59 side view1:29 follow view#CFD #GPU #FluidX3D #OpenCL
X-wing - largest single-GPU CFD simulation ever done (AMD MI250) Dr. Moritz Lehmann 2022-07-28 | Watch out, Skywalker! The lasers on your X-wing create turbulence!TIE fighter simulation: youtu.be/CluQBy51mh8FluidX3D source code: github.com/ProjectPhysX/FluidX3DThis is the largest CFD simulation ever done on a single GPU, cracking 1.25×10⁹ LBM grid points with FluidX3D on the mighty AMD Instinct MI250 (only on 1 GCD with 64GB*).Simulating 50k time steps took 104 minutes at 1076×2152×538 resolution, plus 40 minutes for rendering 4x 30s 1080p video. Shown is the Q-criterion isosurfaces with marching-cubes. Reynolds number is 100k with Smagorinsky subgrid model, but at this resolution it would probably also run stable without.*The MI250 is actually 2 entirely separate GPUs (GCDs) in a single socket with 64GB memory each. One GCD can't directly access the memory of the other. This simulation is only running on one GCD, using the full 64GB of its unified memory.To use both GCDs, or better all 8 GCDs in the server at Jülich Supercomputing Center, the code would need to be specifically extended for multi-GPU support. This is very difficult and time-consuming, and for some parts of FluidX3D, like raytracing graphics, it is close to impossible. Maybe one day I will find the time to do it, but not now.Still, how is it possible to squeeze 1.25 billion grid points in only 64GB?I'm using two techniques here, which together form the holy grail of lattice Boltzmann, cutting memory demand down to only 55 Bytes/node for D3Q19 LBM, or 1/3 of conventional codes:1. In-place streaming with Esoteric-Pull. This almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries.Paper: doi.org/10.3390/computation100600922. Decoupled arithmetic precision (FP32) and memory precision (FP16): all arithmetic is done in FP32, but LBM density distribution functions in memory are compressed to FP16. This almost cuts memory demand in half and almost doubles performance, without impacting overall accuracy for most setups.Paper: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsGraphics are done directly in FluidX3D with OpenCL, with the raw simulation data already residing in ultra-fast video memory. No volumetric data (1 frame of the velocity field is 14GB!) ever has to be copied to the CPU or hard drive, but only rendered 1080p frames (8MB) instead. Once on the CPU side, a copy of the frame is made in memory and a thread is detached to handle the slow .png compression, all while the simulation is already continuing. At any time, about 16 frames are compressed in parallel on 16 CPU cores, while the simulation is running on GPU.Paper: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLTimestamps:0:00 bottom up view0:30 follow view0:59 side view1:29 top down view#CFD #GPU #FluidX3D #OpenCL
Commercial CFD would take months for this | its just minutes with FluidX3D Dr. Moritz Lehmann 2022-07-09 | This is a quick FluidX3D CFD simulation of the Boeing 757 at Re=100k. 912×1824×456 grid resolution, 100k time steps, 2h54min compute+render time (~80min compute, ~94min render) on an A100 40GB GPU.Timestamps:0:00 grid view1:00 follow view2:00 bottom view2:59 side view3:59 top viewSince the last video I have added Smagorinsky subgrid turbulence model to get turbulence down to the resolution limit while still having a stable simulation.Rendering: velocity-colored Q-criterion at Q=0.00005.I wrote the FluidX3D software from scratch during the last 4 years. Here it's standard lattice Boltzmann method (LBM), D3Q19, BGK collision, mid-grid bounce-back boundaries. For the box walls I use equilibriium boundaries. My implementation is ultra (memory) efficient with Esoteric-Pull streaming and FP16 memory compression, reducing memory demand to 55 Bytes/node, 1/3 of other FP32 implementations and almost doubling performance. This way I can get extreme resolution even on a single GPU with limited memory capacity.Compute efficiency on the Nvidia A100 40GB is ~79% at 16000 MLUPS/s.Esoteric-Pull: doi.org/10.3390/computation10060092FP32/FP16 mixed precision: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsFluidX3D: doi.org/10.15495/EPub_UBT_00005400#FluidX3D #OpenCL #CFD
How aerodynamic is Boeing 747 at Re=50k? (Nvidia A100) Dr. Moritz Lehmann 2022-07-03 | This is a quick FluidX3D CFD simulation of the Boeing 747. Re=50k, eye-watering 912×1824×456 resolution, 100k time steps, 6h34min compute+render time on an A100 40GB GPU.Rendering: velocity-colored Q-criterion at Q=0.00002.My FluidX3D code is standard lattice Boltzmann method (LBM), D3Q19, BGK collision, simple mid-grid bounce-back boundaries. For the box walls I use equilibriium boundaries.My implementation is ultra (memory) efficient with Esoteric-Pull streaming and FP16 memory compression, reducing memory demand to 55 Bytes/node, 1/3 of other FP32 implementations. This way I can get extremely large resolution even on a single GPU with limited memory capacity.Esoteric-Pull: doi.org/10.3390/computation10060092FP32/FP16 mixed precision: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsFluidX3D: doi.org/10.15495/EPub_UBT_00005400#FluidX3D #OpenCL #CFD
How aerodynamic is Stanford bunny? Dr. Moritz Lehmann 2022-06-15 | I swear this is essential to my PhD research...Real time #CFD simulation on Nvidia GTX 960M #GPU with my #FluidX3D software.
Running Cyberpunk 2077 in Command Prompt using ASCII Dr. Moritz Lehmann 2022-05-29 | Video(games) in cmd.exe - this is a fun little C++ project, inspired by the legendary 8088 Corruption/Domination programs. Windows cmd.exe can only display standard ASCII characters in 16 colors. Using a 12x7 pixel font and shading characters 176 ░, 177 ▒ and 178 ▓, I can increase the number of colors to 616 using dithering. Contiguous characters of the same colors are combined into blocks to minimize the number of color change commands.Optimizations to improve framerate: - lookup table for color conversion instead of doing nearest-neighbor-search algorithm just-in-time - compare downsampled frame with previous frame and only draw the pixels/characters that have changed; draw changed characters in contiguous segments to minimize std::cout and color change commands - framerate is now between 40% to 400% higher (best for scenes with stationary background)PC specs: - CPU: Intel Core i7-8700K (6C/12T ~4.3GHz) - GPU: Nvidia Titan Xp (3840C ~1.8GHz) - RAM: 16GB DDR4 3200MT/s - SSD: Samsung 970 Pro NVME (3500MB/s read, 2700 MB/s write)Demo with some movie scenes: youtu.be/tehGAbRB-swDemo on Windows Subsystem for Linux, streaming video through SSH from a remote server: youtu.be/72VGIiWKMEUCode Snippets:- Colors in Console with C++: stackoverflow.com/a/67195569/9178992- Dithering: stackoverflow.com/a/68460068/9178992- Screen Recording in C++: stackoverflow.com/a/68483943/91789928088 Corruption/Domination- youtu.be/L6CkYou6hYU- youtu.be/MWdG413nNkITimestamps:0:00 launching Cyberpunk.exe0:16 entering UAV with Rogue2:21 taking down Juroujin satellite3:27 crashing in Arasaka Tower15:02 Adam Smasher fight19:43 last mission
Combined scientific CFD simulation and interactive raytracing with OpenCL (IWOCL & SYCLcon 2022) Dr. Moritz Lehmann 2022-05-28 | This is my IWOCL & SYCLcon 2022 technical talk.Extended Abstract: researchgate.net/publication/360501260_Combined_scientific_CFD_simulation_and_interactive_raytracing_with_OpenCLSlides: iwocl.org/wp-content/uploads/06-presentation-iwocl-syclcon-2022-lehmann.pdf0:00 Intro & Demos4:16 Basic Idea6:54 Lattice Boltzmann CFD8:35 Benchmarks & Efficiency11:33 Application12:37 Graphics with OpenCL23:06 Conclusions24:16 More Demos!!26:09 OutroOne of the main uses for OpenCL is compute applications where rendering is done externally. However separating simulation and rendering has many disadvantages, especially the slowdown caused by copying data from device to host, and needing to store raw data on the hard drive.A much faster approach is to implement both simulation and rendering in OpenCL. The rendering kernels have direct read-only access to the raw simulation data in ultra-fast GPU memory. This eliminates almost all PCIe data transfer, allowing for interactive visualization in real time while the simulation is running.Although OpenCL does not have existing functionality for graphical rendering, being a general compute language, it allows for implementing an entire graphics engine. On top, specific low-level optimizations make this OpenCL graphics engine outperform any existing rendering solution for this scenario, enabling fluid raytracing in real time on even non-RTX GPUs.This is demonstrated with the software FluidX3D [1].First I introduce the lattice Boltzmann method for simulating physically accurate fluid flow and discuss optimizations: Being a memory-bound algorithm, coalesced memory access is key. This is achieved through AoS data layout and the one-step-pull LBM streaming scheme. One-step-pull leverages that misaligned read penalty is much smaller than misaligned write penalty on most GPUs. Roofline analysis shows that with these optimizations, the LBM runs at 100% efficiency on the fastest GPUs [2].For free surface flows, the LBM is extended with an efficient Volume-of-Fluid (VoF) implementation [3]. This allows covering new grounds in science: FluidX3D has been used to simulate more than 1600 raindrop impacts to evaluate how microplastics transition from the ocean surface into the atmosphere [4].Rasterization on the GPU is parallelized not over pixels but lines/triangles instead. Each line/triangle is transformed from 3D to 2D screen coordinates and then rasterized onto the frame (integer array) with Bresenham [5] and z-buffer.The raytracing graphics are based on a combination of fast ray-grid traversal and marching-cubes, leveraging that the computational grid from the LBM already is an ideal acceleration structure for raytracing. The idea is simple: Through each pixel on screen, shoot a reverse light ray out of the camera and see where it intersects with a surface in the scene. Then (recursively) calculate reflected/refracted rays and mix colors. If a ray doesn't intersect anything, its color is determined by the skybox.With mesh surfaces consisting of triangles, computation time quickly becomes a problem, as all triangles have to be tested for intersection. To overcome this, an acceleration structure is required. While games often use a bounding volume hierarchy, the LBM already provides an ideal alternative acceleration structure: the simulation grid. The corresponding algorithm is called ray-grid traversal: When a ray shoots through the 3D grid, intersections with the surface only have to be checked for at each traversed grid cell rather than the entire grid. Triangles are generated on-the-fly with marching-cubes and ray-triangle intersections are checked with Möller-Trumbore. If an intersection has been found, the normals are calculated on the 8 grid points spanning the cell, and are trilinearly interpolated to the intersection. The interpolated surface normal makes the raytraced surface appear smooth.On the GPU, the ray(s) for each pixel are computed in parallel. It is important how to align the OpenCL workgroups: best performance is achieved for 8x8 pixel tiles; this is about 50% faster than 64x1 tiles, because all rays of the workgroup are more likely to traverse the same grid cells, improving memory broadcasting. For marching-cubes, the algorithm by Paul Bourke [6] is implemented in OpenCL. Table size is reduced to 1/8. The Möller-Trumbore algorithm [7] is implemented branchless.This raytracing implementation is fast enough to run in real time for even the largest lattice dimensions that fit into GPU memory. This is demonstrated on the most realistic simulation of an impacting raindrop ever done [8].[1] doi.org/10.15495/EPub_UBT_00005400[2] arxiv.org/abs/2112.08926[3] doi.org/10.3390/computation10020021[4] doi.org/10.1186/s43591-021-00018-8[5] doi.org/10.1147/sj.41.0025[6] http://paulbourke.net/geometry/polygonise[7] doi.org/10.1080/10867651.1997.10487468[8] youtu.be/1g_zFsvScME
Falling Ring Splash Dr. Moritz Lehmann 2022-05-15 | I recreated the initial phase of the disk splash in CFD with my FluidX3D OpenCL lattice Boltzmann code. Video of the experiment can be found here: reddit.com/r/oddlysatisfying/comments/um3zo0/water_cake_by_wmhejiuGrid resolution is 560³, and parameters match the experiment 1:1. I estimated geometry via the centimeter-grid on the table, fall height 0.5m, viscosity / density / surface tension of water.The disk in the simulation moves; I use moving bounce-back and periodically update the flag lattice. This was simulated on my desktop PC (Titan Xp GPU) and took 78 minutes. With a novel "Esoteric-Pull" in-place streaming scheme, combined with FP32/FP16 mixed precision, the memory requirements for the Volume-of-Fluid lattice Boltzmann method is cut down from 181 to 67 Bytes/Node, about 1/3 of vanilla LBM. This enables colossal resolutions.Raindrop simulations: doi.org/10.1186/s43591-021-00018-8Volume-of-Fluid: doi.org/10.3390/computation10020021Esoteric-Pull: doi.org/10.3390/computation10060092FP32/FP16 mixed precision: researchgate.net/publication/362275548_Accuracy_and_performance_of_the_lattice_Boltzmann_method_with_64-bit_32-bit_and_customized_16-bit_number_formatsFluidX3D: doi.org/10.15495/EPub_UBT_00005400The simulation software used is FluidX3D, an OpenCL implementation of the lattice Boltzmann method. FluidX3D runs at peak hardware efficiency on the worlds fastest data-center GPUs (8799/5232 MLUPs/s on the Nvidia A100/V100 @ D3Q19 SRT FP32), on gaming GPUs and CPUs and it even runs my smartphone.The rendering is done by FluidX3D as well, now with raytracing. Simulated light rays from the camera traverse the LBM lattice and check for intersections with the triangles that marching-cubes generates on-the-fly from the isovalues. Triangles never leave registers.The 3D lattice that the LBM provides already is one of the ideal acceleration structures for raytracing. Any ray only has to check for possible intersections with iso-surface triangles contained within the traversed grid cells. This makes it - when the lattice is not too large - even run in real time on non-RTX GPUs.FluidX3D is now built entirely on my open source OpenCL-Wrapper, reducing the entire code on the C++ side to about half. If you want to get into OpenCL yourself, look no further: github.com/ProjectPhysX/OpenCL-Wrapper#FluidX3D #OpenCL #Raytracing #GPU #LBM
