SAP: Multiscale Methods for FSI

SAP Project:

Multiscale Methods for Fluid-Structure Interaction and Modeling of Turbulence on Moving Grids

Arif Masud

Civil and Materials Engineering Department

University of Illinois at Chicago

Research Objectives
SCIENTIFIC GOALS
Fluid-structure interaction (FSI) problems belong to the class of multiphysics problems that appear in several engineering disciplines. The current computational methods work reasonably well in predicting coupled effects in mildly nonlinear FSI problems. However, mathematically non-smooth FSI problems pose a great challenge to the current computational techniques and call for the development of mathematically rigorous formulations, and robust and efficient solution algorithms. One such problem that is relevant in navy applications is that of fluid-structure interaction around propellers of submarines and surface ships (see Figure 1). Likewise, an example from biofluid dynamics is that of the modeling of blood-flow through a pulsating heart, and the blood artery-wall interaction. Multiscale mathematical formulations that can account for the multiple spatial, temporal and material length scale, along with efficient staggered solution algorithms that inherit the properties of the underlying coupled multiphysics problems, are required to successfully address the class of non-smooth FSI problems.

Figure 1. Configuration of a typical submarine

COMPUTATIONAL GOALS AND METHODS
Multiphysics problems can be characterized as (i) interacting problems that have two or more geometrically distinct regions that interact at their boundaries through the transfer of motion, forces and heat, and (ii) coupled problems that have geometric regions in which behavior is characterized by simultaneous physical phenomena, some or all of which depend on one another. FSI problems are, in fact, a combination of both characterizations, and as such, they are intellectually stimulating, mathematically challenging, and computationally intensive. Coupled analysis of FSI problems combines fluids, solids and their interactions in a single-pass simulation with the intent of capturing the true multiphysics behavior of the system. Since transient FSI calculations are computationally intensive, a robust capability requires a comprehensive array of features, namely: (i) mathematical formulations that are stable and convergent, and account for the scale effects, (ii) efficient nonlinear solution techniques, (iii) domain decomposition techniques, and (iv) automatic mesh refinement strategies. In addition the computational code needs to be able to incorporate the modern computational advancements in terms of scalability of the code. If these factors are not accounted for in the design phase of the coupled solution schemes the ensuing mathematical and computational complexities can render the resulting solution strategies only conditionally stable. When such algorithms are applied to large-scale computations, a clear understanding of the stability of the algorithm, or the accuracy of the solution, is lost.

The goal of this effort is to extend the recently developed multiscale finite element methods for fluids and solids to large-scale 3D computations over moving meshes. We will use the problem of FSI around propellers as our driving model problem for the development and validation of the multiscale finite element technology (see the animation). This problem involves large rotations and sliding of meshes, and turbulence that is generated over rotating hydrofoils, which is then transported downstream over time-dependent meshes. To obtain physically meaningful solution to this problem, one needs to be able to model turbulence over moving, and possibly deforming, meshes. In addition, large-scale computational resources are required to simulate the real problems and to validate the computational method. Development of such a computational capability for non-smooth FSI problems is the prime goal of this effort.

POTENTIAL BENEFITS
The exponential growth of computer speed and capacity over the past two decades has led to the development of computational science and engineering as a unique and powerful tool for simulating and visualizing the behavior of complex engineering systems. Expected increased accessibility within the next five years by the national scientific and engineering communities to computing systems whose performance is approximately one thousand times greater than the computers now widely available will make computer-based simulation, design, and optimization a routine job for engineers. For general nonlinear FSI systems where a time accurate simulation is the only authentic approach, high-fidelity computer programs and computer algorithms will be required to take full advantage of the opportunities offered by the developments in hardware. The present project addresses one such area in the field of hydrodynamic (aerodynamic) shape optimization of engineering objects and vehicles that are subjected to given hydrodynamic (aeroelastic) constraints. It will thus be possible to obtain accurate and economic solution to the multi-disciplinary design problem for the various design variables in the entire spectrum of the design space, leading to a true simulation-based design paradigm.

We believe that the proposed efforts on non-smooth FSI problems will substantially enhance the current state-of-the-art in multiphysics simulation of FSI problems. We envision that these developments will have tremendous application in the area of biofluid dynamics as well.

COMPUTATIONAL APPROACH
The computational approach being envisioned here will have several ingredients. The fluid mechanics formulation will be based on the extension to 3D of our multiscale and stabilized finite element methods for the incompressible Navier-Stokes equations. The formulations employed to model the structure will be based on the variational multiscale method that accounts for the material length scales in a unified way. Similarly, the mesh moving schemes will be based on the extension to 3D of our mesh rezoning method. Since the intended computations will invariably involve large and dense meshes, we will employ the parallel preconditioned iterative solvers as well as the parallel direct solvers available at the National Center for Supercomputing Applications (NCSA). This project will also help us in investigating the design of a new matrix solver that is based on the notion of domain decomposition, together with the ideas of solution separation into "interior" and "boundary" regions. Such a parallel matrix solver can be of tremendous help in the solution of large matrices typically encountered in finite element computations.

Visualization of large amounts of data will necessitate the use of Paraview™ software as well as other visualization capabilities available at NCSA.

ACCOMPLISHMENTS AND SIGNIFICANCE
We have successfully developed multiscale finite element methods for incompressible Navier-Stokes equations, and convective diffusive heat transfer problems [1-4]. Multiscale/stabilized methods have also been developed for linear and nonlinear solid mechanics [5,6]. Multilayered material systems undergoing elastoplastic deformations are considered [7,8]. Coupled interaction problems have been addressed [9–12]. An essential ingredient of the overall FSI strategy is an adaptive mesh rezoning scheme, which has been developed [9, 10, 13]. We have now extended the multiscale formulation for incompressible Navier-Stokes equations to 3D, and an intended extension is to account for the effects of turbulence in computations over moving and deforming meshes.

PUBLICATIONS
1. A. Masud, "On a stabilized formulation for incompressible NavierStokes equations", Proc. 6th Japan-US International Symposium on Flow Simulation and Modeling, (ed. Hiroshi Kanayama), Fukuoka, Japan. pp. 33-38, May 2002.

2. M. Ayub and A. Masud, "A new stabilized formulation for convective-diffusive heat transfer", Numerical Heat Transfer, Part-B, Vol. 44, pp. 1-23, 2003.

3. A. Masud and R.A. Khurram, "A multiscale/stabilized finite element method for the advection diffusion equation", Computer Methods in Applied Mechanics and Engineering, 193, pp. 1997-2018, 2004.

4. A. Masud and R.A. Khurram, “A multiscale finite element method for the incompressible Navier-Stokes equation”, Computer Methods in Applied Mechanics and Engineering , In Press, 2005.

5. A. Masud and K. Xia, “A stabilized mixed finite element method for nearly incompressible elasticity”, Journal of Applied Mechanics, Vol. 72, September 2005.

6. A. Masud and K. Xia, “A variational multiscale method for computational inelasticity: Application to superelasticity in shape memory alloys”, Computer Methods in Applied Mechanics and Engineering , In Press, 2005.

7. A. Masud and C.L. Tham, “A 3-D co-rotational framework for finite deformation elasto-plastic analysis of multilayered composite shells”, AIAA Journal, 38, No. 12, pp. 2320-2327, 2000.

8. C.L. Tham, Z. Zhang and A. Masud, “An elastoplastic damage model cast in a corotational kinematic framework for large deformation analysis of laminated composite shells”, Computer Methods in Applied Mechanics and Engineering , 194/21-24, pp. 2641-2660, 2005.

9. A. Masud, “A Space-time Finite Element Method for Fluid-structure Interaction”, PhD Thesis, Stanford University, April 1993 .

10. A. Masud and T.J.R. Hughes, “A space-time Galerkin/least-squares finite element formulation of the Navier-Stokes equations for moving domain problems”, Computer Methods in Applied Mechanics and Engineering, 146, pp. 91-126, 1997 .

11. A. Masud, R.A. Khurram and M. Bhanabhagvanwala, “A stable method for fluid-structure interaction problems”, Proceedings CD-ROM of the Sixth World Congress on Computational Mechanics, (eds. Z.H. Yao, M.W. Yuan and W.X. Zhong). ISBN 7-89494-512-9, Beijing, China, 2004.

12. R.A. Khurram and A. Masud, “Multiscale/Stabilized Incompressible Navier-Stokes Equations for Fluid-Structure Interaction”, Computational Mechanics, In press, 2005.

13. A. Masud, M. Bhanabhagvanwala and R.A. Khurram, “An Adaptive Mesh Rezoning Scheme for Moving Boundary Flows and Fluid-Structure Interaction”, Computers and Fluids, In Press, 2005.

Status Report
April 4.2006

Conversion of scalar code to parallel
The timing study of the code revealed that the code is spending around 90% of the wall time in the solver. This is typical for direct solvers for incompressible Navier-Stokes solution with mixed pressure-velocity formulation. As a first-cut, coarse-grain parallelization is done, the solver part from the scalar code is parallelized by employing SGI PSLDU solver. These routines are part of the SCSL Scientific Library and can be loaded using either the -lscs or the -lscs_mp option. The -lscs_mp option directs the linker to use the multiprocessor version of the library. PSLDU solves sparse unsymmetrical linear systems of the form Ax = b where A is an n-by-n input matrix having symmetric nonzero pattern but unsymmetrical nonzero values, b is an input vector of length n, and x is an unknown vector of length n. PSLDU uses a direct method. A is factored into the following form:

A = L D U

where L is a lower triangular matrix with unit diagonal, D is a diagonal matrix, and U is an upper-triangular matrix with unit diagonal. Sparse matrix A must be input to PSLDU in Harwell-Boeing format (also known as Compressed Column Storage format). A mapping routine is included in the 3D Navier-Stokes code to convert the global matrix into Harwell-Boeing format.
Timing of the parallel part
A study is conducted where timing is recorded for various steps of parallel part of the code. Following are the steps involved in parallelization:

Mapping of the global matrix in HB format
Ordering
Preprocessing
Factorization
Solution

Table 1.1. Wall times for various steps of direct parallel solver for a 62,500 x 62,500 matrix

Step	Wall time (seconds)
Mapping	45.130858 (13 %)
Ordering	0.00098 (0 %)
Preprocessing	105.94045 (30 %)
Factorization	204.1054 (57 %)
Solution	3.5918 (1 %)

Table 1.2. Wall times for various steps of direct parallel solver for a 108,000 x 108,000 matrix

Step	Wall time (seconds)
Mapping	167.8253 (11 %)
Ordering	0.001 (0 %)
Preprocessing	395.6172 (25 %)
Factorization	1005.0945 (64 %)
Solution	13.032 (1 %)

Scaling study
A mesh of 50 x 50 x 50 (500,000 x 500,000 global matrix) is run with 20, 40 and 80 processors. The wall times obtained for one iteration (i.e., one direct solve) is given in Table 1.3.

Table 1.3. Scaling study of PLSLDU of direct parallel solver for a 500,000 x 500,000 matrix


No. of processors	Wall time
20	11 hrs. 48 min.
40	10 hrs. 42 min.
80	9 hrs. 39 min.

The time it takes in various phases of PSLDU can be estimated from Tables 1.1 and 1.2. As the problem size increases, the percentage of time for factorization increases, and the times for other phases decrease. This can be explained by the fact that factorization is a stronger function of problem size (O[n³]) than other phases (O[n²]). The percentage of factorization time for problem size, n = 500,000, would be around 70 - 80%. There are too many parameters involved in this extrapolation, and these are crude approximations. Corresponding times are shown in Table 1.4.

Table 1.4. Estimated time in factorization phase for a 500,000 x 500,000 matrix

No. of processors	Estimated wall time
20	8 hrs. 51 min.
40	8 hrs. 02 min.
80	7 hrs. 14 min.

Comments
Most of the time is spent in the preprocessing and the factorization phases. The mapping is not the bottleneck. The scaling study reveals that parallelization is not optimal, which could be because of the nature of the matrix in this study. Incompressible Navier-Stokes equation in mixed pressure-velocity formulation yields an ill-conditioned matrix with very small diagonal values, corresponding to pressure equation. This requires massive partial pivoting, which normally results in poor parallel performance of LU decomposition. This problem could be resolved by repeating this study for a same-sized matrix representing different physical problems (e.g., heat conduction or elasticity) where partial pivoting is not a major issue. Once the parallel efficiency of PSLDU solver is established, different ideas, like static condensation of pressure or solving pressure Poisson equation can be employed for incompressible Navier-Stokes equations. Different direct and iterative solvers will also be included in the future study. The parallel efficiency can also be improved by decreasing the scalar work (i.e., parallelizing other time consuming steps like assembly and element matrix generation).