GPUs TO MARS

GPUs TO MARS Full Scale Simulation of SpaceX’s Mars Rocket Engine Adam Lichtl, Stephen Jones, GTC 2015

Background  Independent space launch company  Build and operate our own rocket & capsule  Operations in CA, TX & FL  Founded in 2002, first launch in 2008  18 successful missions to date

Design

Build

Launch

Mars Length of day 24 hours, 40 minutes Length of year 687 days Gravity 0.375g s Distance from Sun 1.53 AU Mars Air CO (96%), N (2%) 2 2 Earth Air O2 (21%), N2 (78%) Atmosphere 1% density / pressure of Earth Temperature -284 / +86 / -81 F (low / high / avg)

Human Mars Mission Classes Credit: CollectSpace

Mass to Mars Apollo Command & International Space Station NASA Mars Architecture Service Module  Mass: 450 tons  Mass: 300 tons  Mass: 46 tons  36x Space Shuttle,  3x heavy-lift  Terrestrial assembly 5x Proton launches launches  In-orbit assembly Image Credit: NASA Image Credit: NASA Image Credit: Mark Benson & Kerbal Space Program

Methane Fuel Carbon dioxide from atmosphere Synthesize return-journey fuel on Mars O 2H O + CO -> CH + 2O O C O C 2 2 4 2 O O H C H O O H H O Water from below ground H O O H H H

Rocket Engine Design

Why Simulation? 1. Investigate what cannot be measured 2. Reduce need for testing 3. Design optimisation: narrow design space 4. Proactive instead of reactionary design

Time scales vary by 8 orders of magnitude Diffusion Advection Acoustics -11 -10 -9 -8 -7 -6 -5 -4 -3 10 10 10 10 10 10 10 10 10 seconds Reaction Chamber Residence

Injection Length scales vary by 6 orders of magnitude log E(k) Cascade y Combustion Chamber: ~1m Energy Edd Kolmogorov Scale: ~1µm Viscous dissipation kmin Eddy Wavenumber kmax 3 6 18 Total points: (10 ) = 10 = 1,000,000,000,000,000,000 points @1kB per point -> Yottabytes of data

Structure in Turbulence Credit: SpaceWeather

Simulation Meshing Structured Grid Adaptive Grid Unstructured Grid

Fourier Transform: Spectral Compression

Wavelets: Local Fractal Basis

Wavelet Compression

Wavelet Compression δ δ δ > ε δ < ε (ε is some tolerance)

Wavelet Compression Transform Output

Decompression Transform Output

Decompression +δ Transform Output

Decompression Transform Output

Regular Grid: Direct Indexing (x, y, z) (x, y+1, z) Index (x, y, z) = p Index (x, y+1, z) = p + Nx

Sparse Grid: Difficult Indexing (x, y, z) (x, y+1, z) Index (x, y, z) = p Index (x, y+1, z) = ???

Sparse Grid Point Lookup Linear search Tree-based search Hybrid tree search + dense indexing

Grid Point Contains Reference to Data Grid Point 14 Location (x,y,z) Data Reference (x, y, z)

Grid Point Contains Reference to Data Grid Point 14 Grid Index 14 15 16 Location (x,y,z) (11,11,5) (11,12,5) (Location) Grid Data Data Reference 3 8 (Reference) (x, y, z)

Grid Point Contains Reference to Data Grid Point 14 Grid Index 14 15 16 Location (x,y,z) (11,11,5) (11,12,5) … Data Reference Grid Data 3 8 … density ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9 (x, y, z) X-momentum ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9 Y-momentum ρv ρv ρv ρv ρv ρv ρv ρv ρv ρv 0 1 2 3 4 5 6 7 8 9 Z-Momentum ρw ρw ρw ρw ρw ρw ρw ρw ρw ρw 0 1 2 3 4 5 6 7 8 9 Energy E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Simulation Data

Grid Adaptation Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 Reference 0 1 2 3 4 5 6 7 8 9

Grid Adaptation Grid Index P0 P2 P4 P6 P8 Location x y x y x y x y x y 0 0 2 2 4 4 6 6 8 8 Reference 0 2 4 6 8

Grid Adaptation Grid Index P0 P1 P2 P3 P4 Location x y x y x y x y x y 0 0 2 2 4 4 6 6 8 8 Reference 0 2 4 6 8

Grid Transformations Track Data Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 Reference 0 1 2 3 4 5 6 7 8 9

Grid Transformations Track Data Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 Reference 0 1 2 3 4 5 6 7 8 9 Delete Add Sort Transpose

Grid Transformations Track Data Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 Reference 0 1 2 3 4 5 6 7 8 9 Delete Add Sort Transpose Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y 4 4 7 7 9 9 0 0 5 5 1 1 8 8 3 3 6 6 2 2 Reference 4 7 9 0 5 1 8 3 6 2

Losing Coalesced Memory Access Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Reference 0 1 2 3 4 5 6 7 8 9 ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9 ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9 ρv ρv ρv ρv ρv ρv ρv ρv ρv ρv 0 1 2 3 4 5 6 7 8 9 ρw ρw ρw ρw ρw ρw ρw ρw ρw ρw 0 1 2 3 4 5 6 7 8 9 E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Coalesced Simulation Data

Losing Coalesced Memory Access Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Reference 4 7 9 0 5 1 8 3 6 2 ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9 ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9 ρv ρv ρv ρv ρv ρv ρv ρv ρv ρv 0 1 2 3 4 5 6 7 8 9 ρw ρw ρw ρw ρw ρw ρw ρw ρw ρw 0 1 2 3 4 5 6 7 8 9 E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Uncoalesced Simulation Data

Restoring Coalesced Memory Access Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y 4 4 7 7 9 9 0 0 5 5 1 1 8 8 3 3 6 6 2 2 Reference 4 7 9 0 5 1 8 3 6 2 ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9 ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9 ρv ρv ρv ρv ρv ρv ρv ρv ρv ρv 0 1 2 3 4 5 6 7 8 9 ρw ρw ρw ρw ρw ρw ρw ρw ρw ρw 0 1 2 3 4 5 6 7 8 9 E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 Uncoalesced Simulation Data

Restoring Coalesced Memory Access Grid Index P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Location x y x y x y x y x y x y x y x y x y x y x y x y x y x y x y x y x y x y x y x y 4 4 7 7 9 9 0 0 5 5 1 1 8 8 3 3 6 6 2 2 4 4 7 7 9 9 0 0 5 5 1 1 8 8 3 3 6 6 2 2 Reference 4 7 9 0 5 1 8 3 6 2 Reset Indices 0 1 2 3 4 5 6 7 8 9 ρ0 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ9 ρ4 ρ7 ρ9 ρ0 ρ5 ρ1 ρ8 ρ3 ρ6 ρ2 ρu0 ρu1 ρu2 ρu3 ρu4 ρu5 ρu6 ρu7 ρu8 ρu9 ρu4 ρu7 ρu9 ρu0 ρu5 ρu1 ρu8 ρu3 ρu6 ρu2 ρv ρv ρv ρv ρv ρv ρv ρv ρv ρv Reorder ρv ρv ρv ρv ρv ρv ρv ρv ρv ρv 0 1 2 3 4 5 6 7 8 9 Simulation 4 7 9 0 5 1 8 3 6 2 ρw ρw ρw ρw ρw ρw ρw ρw ρw ρw Data ρw ρw ρw ρw ρw ρw ρw ρw ρw ρw 0 1 2 3 4 5 6 7 8 9 4 7 9 0 5 1 8 3 6 2 E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 E4 E7 E9 E0 E5 E1 E8 E3 E6 E2 Uncoalesced Simulation Data Coalesced Simulation Data

Coupled Multi-Physics Advection Chemistry Thermodynamics Diffusion

Real Gas Properties Extreme pressure requires real-gas properties, evaluated at each grid node  Viscosity Pressure  Conductivity  Diffusion constants  Per-species diffusion rates Independent Calculations Massively Parallelizable Temperature O phase diagram, Z-axis is density 2 (colour: liquid is blue, gas is white)

Strong Temperature Dependence Arrhenius equation includes temperature in exponent

Chemical Kinetic Model H H + O O O H H O H H H H H O H O 2 2 2

Chemical Kinetic Model H H + O X O O H H O H H H H H O H O 2 2 2 O H H O OH O O H O H O O 2 2 O H H HO H 2

Chemical Kinetic Model H H + O O O H H O H H H H H O H O 2 2 2 O H H O OH O O H O H O O 2 2 O H H HO H 2

Methane Chemistry H H O O O C + O O C + H H O H H O O H H CH O CO H O 4 2 2 2

Methane Chemistry 53 Species, 325 Reactions H H O O O C + O O C + H H O H H O O H H CH O CO H O 4 2 2 2

(Actual simulation data, rendered with a ray tracer)

Acknowledgements Oleg Vasilyev, University of Colorado Boulder Department of Mechanical Engineering Jonathan Regele, Iowa State University Department of Aerospace Engineering Don Lamb The Flash Center for Computational Science Praveen Ramaprabhu University of North Carolina at Charlotte Joseph Oefelein, Sandia National Laboratories Combustion Research Facility Marc Massot Ecole Centrale, Paris

Want To Help? If you have a software or physics background and you’re interested in working with us, please contact [email protected]