相场模拟的并行计算方法

HybridMPI+OpenMPApproachtoImprovet[e-mail]19th,2013ProjectBackgroundProjectTeam(UniversityofMichigan):KatsuyoThornton(P.I.),VictorChanPhase-field-crystal(PFC)formulationtostudydynamicsofvariousmetalsystemsOriginalin-housecodewritteninC++Hasbeenrunin2Dand3DsystemsSolvesmultipleHelmholtzequations,areduction,thenanexplicittimestep2SolvingtheHelmholtzEquations𝛻2∅+𝒌𝟐∅=𝟎OriginallyusedGMRESwithAlgebraicMultigrid(AMG)preconditionerfromHYPREIn3D,discretizationmatrixislargeandmaybecomeindefinitedifficulttosolve,requiringlargeiterationsPoorweak-scalingresultsProhibitivelylongforindefinitematrixcaseIncreasingmemoryrequirementswithiteration3GoalScalabletosolvelargerproblem–Weakscaling:maintainthetime-to-solutionwithincreasingnumberofprocessesandafixedproblemsizeperprocessDecreasethetimetosolutionto1sec/timestep–Strongscaling:decreasetime-to-solutionwithincreasingnumberofprocessandafixedproblemsize–Exploitotherparallelism(withOpenMP?)–Investigatebetterpreconditioner–Differentmethod(library?)tosolvetheequations4GoalScalabletosolvelargerproblem–Weakscaling:maintainthetime-to-solutionwithincreasingnumberofprocessesandafixedproblemsizeperprocessDecreasethetimetosolutionto1sec/timestep–Strongscaling:decreasetime-to-solutionwithincreasingnumberofprocessandafixedproblemsize–Exploitotherparallelism(withOpenMP?)–Investigatebetterpreconditioner–Differentmethodtosolvetheequations5ComplexIterativeJacobiSolverHadley,G.R,AcomplexJacobiiterativemethodfortheindefiniteHelmholtzEquation,J.Comp.Phys.203(2005)358-370ReplacedHYPREAmodificationofstandardJacobimethod𝐻𝑛+1𝐻𝑛,Δ𝑙𝑖,𝛿𝑖2,𝛿𝑖2iscomputedwithcentered-differenceEasilyparallelizedandlowmemoryrequirementConvergenceratedependsonresolution,butroughlyconstantfromproblemtoproblemlargerproblem(withsimilarresolution)shouldnotincreaseiterations.6ComplexIterativeJacobiSolverHadley,G.R,AcomplexJacobiiterativemethodfortheindefiniteHelmholtzEquation,J.Comp.Phys.203(2005)358-370ReplacedHYPREAmodificationofstandardJacobimethod𝐻𝑛+1𝐻𝑛,Δ𝑙𝑖,𝛿𝑖2,𝛿𝑖2iscomputedwithcentered-differenceEasilyparallelizedandlowmemoryrequirementConvergenceratedependsonresolution,butroughlyconstantfromproblemtoproblemlargerproblem(withsimilarresolution)shouldnotincreaseiterations.7Adraftversionwasquicklyimplementedbytheprojectteam(VictorChan)andtested.ProfilingtheCodewithCrayPATMeasurebeforeoptimizeCanusesamplingortracingUsingCrayPATissimple:loadmodule,re-compile,buildinstrumentedcode,re-runCayPATcantraceonlyspecifiedgroup,e.g.mpi,io,heap,fftw,...moduleloadperftoolsmakecleanmakepat_build–gmpipfc_jacobi.exeaprun–n48pfc_jacobi.exe+patpat_report–oprofile.txt\output_data.xf8ProfilingtheCodewithCrayPATMeasurebeforeoptimizeCanusesamplingortracingUsingCrayPATissimple:loadmodule,re-compile,buildinstrumentedcode,re-runCayPATcantraceonlyspecifiedgroup,e.g.mpi,io,heap,fftw,...moduleloadperftoolsmakecleanmakepat_build–gmpipfc_jacobi.exeaprun–n48pfc_jacobi.exe+patpat_report–oprofile.txt\output_data.xf9ThatShouldHaveWorked!CrayPATWorkaroundUsetheAPIfor“finegrain”instrumentationAdd“PAT_region_{begin/end}”callstomostsubroutinesAfternarroweddowntoacouplemajorsubroutines,splitlabelsto“computation”and“communication”#includepat_api.h...voidComplex_Jacobi(…){...intPAT_ID,ierr;PAT_ID=41;ierr=PAT_region_begin(PAT_ID,communication);MPI_Internal_Communicate(…);MPI_Boundary_Communicate(…)ierr=PAT_region_end(PAT_ID);PAT_ID=42;ierr=PAT_region_begin(PAT_ID,computation);for(inti=1;isize.L1+1;i++){for(intj=1;jsize.L2+1;j++){for(intk=1;ksize.L3+1;k++){residual(i,j,k)=(1.0/D)*(...);}}}ierr=PAT_region_end(PAT_ID);10CrayPATWorkaroundUsetheAPIfor“finegrain”instrumentationAdd“PAT_region_{begin/end}”callstomostsubroutinesAfternarroweddowntoacouplemajorsubroutines,splitlabelsto“computation”and“communication”#includepat_api.h...voidComplex_Jacobi(…){...intPAT_ID,ierr;PAT_ID=41;ierr=PAT_region_begin(PAT_ID,communication);MPI_Internal_Communicate(…);MPI_Boundary_Communicate(…)ierr=PAT_region_end(PAT_ID);PAT_ID=42;ierr=PAT_region_begin(PAT_ID,computation);for(inti=1;isize.L1+1;i++){for(intj=1;jsize.L2+1;j++){for(intk=1;ksize.L3+1;k++){residual(i,j,k)=(1.0/D)*(...);}}}ierr=PAT_region_end(PAT_ID);11CommunicationsubroutineeventuallydominateatcertainMPIsizeCellUpdateandMPICommunication12Stepn:ComputedifferencesandupdatecellvaluesStepn:Communicateupdatedvaluetoneighboringghostcells(usingMPI_Sendrecv())Stepn+1:ComputedifferencesandupdatecellvaluesIterationsThereisafixedcommunicationcostineveryiteration.Canwehideit?rr+1r+2HidingCommunicationCost13Stepn:PostMPI_Irecv()forghostcells.Computedifferencesandupdatecellvaluesonsurfacecells.Stepn:SendsurfacecellsvaluewithMPI_Isend().Computedifferencesandupdatecellvaluesoninnercells.Stepn+1:Computedifferencesandupdatecellvaluesonsurfacecells.PostMPI_Irecv()forghostcells.Iterationsrr+1r+2HidingCommunicationCost14Stepn:Computedifferencesandupdatecellvaluesonsurfacecells.Pos