White R.E. Computational Mathematics: Models, Methods, and Analysis with MATLAB and MPI
Подождите немного. Документ загружается.
270 CHAPTER 6. HIGH PERFORMANCE COMPUTING
Figure 6.6.2: Send and Receive for Processors
at end with the arrow is receiving data. For example, at times 1, 2, 3 and 4
processor 1 will send to processor 2, receive from processor 2, send to processor
0, receive from processor 0, respectively.
In the heat2dmpi.f code lines 1-17 are the global initialization of the vari-
ables. Lines 18-20 start the multiprocessing, and lines 28-55 execute the explicit
finite di
erence method where only the current temperatures are recorded. In
lines 29-34 the computations for the processor my_rank are done for the h or-
izontal subblock of the s pace grid associated with this processor. Note, the
grid rows are associated with the columns in the array uold and unew. The
communications between the processors, as outlined in Figure 6.6.2 for p = 8
processors, is executed in lines 35-54. In particular, processor 1 communications
are done in lines 39—44 when my_rank = 1. After the last time step in line 55,
lines 56-63 gather the computations from all the processors onto processor 0;
this could have been done by the subroutine mpi_gather().
MPI/Fortran Code heat2dmpi.f
1. program heat
2. implicit none
3. include ’mpif.h’
4. real, dimension(2050,2050):: unew,uold
5. real :: f,cond,dt,dx,alpha,t0, timef,tend
6. i nteger :: my_rank,p,n,source,dest,tag,ierr,loc_n
7. integer :: status(mpi_status_size),bn,en,j,k
8. integer :: maxk,i,sbn
9. n = 2049
10. maxk = 1000
11. f = 1000.0
12. cond = .01
13. dt = .01
14. dx = 100.0/(n+1)
© 2004 by Chapman & Hall/CRC
6.6. MPI AND 2D MODELS 271
15. alpha = cond*dt/(dx*dx)
16. uold = 0.0
17. unew = 0.0
18. call mpi_init(ierr)
19. call mpi_comm_rank(mpi_comm_world,my_rank,ierr)
20. call mpi_comm_size(mpi_comm_world,p,ierr)
21. loc_n = (n-1)/p
22. bn = 2+(my_rank)*loc_n
23. en = bn + loc_n -1
24. call mpi_barrier(mpi_comm_world,ierr)
25. if (my_rank.eq.0) then
26. t0 = timef()
27. end if
28. do k =1,maxk
29. do j = bn,en
30. do i= 2,n
31. unew(i,j) = dt*f + alpha*(uold(i-1,j)+uold(i+1,j)&
+ uold(i,j-1) + uold(i,j+1))&
+ (1- 4*alpha)*uold(i,j)
32. end do
33. end do
34. uold(2:n,bn:en)= unew(2:n,bn:en)
35. if (my_rank.eq.0) then
36. call mpi_recv(uold(1,en+1),(n+1),mpi_real, &
my_rank+1,50, mpi_comm_world,status,ierr)
37. call mpi_send(uold(1,en),(n+1),mpi_real, &
my_rank+1,50, mpi_comm_world,ierr)
38. end if
39. if ((my_rank.gt.0).and.(my_rank.lt.p-1) &
.and.(mod(my_rank,2).eq.1)) then
40. call mpi_send(uold(1,en),(n+1),mpi_real, &
my_rank+1,50, mpi_comm_world,ierr)
41. call mpi_recv(uold(1,en+1),(n+1),mpi_real, &
my_rank+1,50, mpi_comm_world,status,ierr)
42. call mpi_send(uold(1,bn),(n+1),mpi_real, &
my_rank-1,50, mpi_comm_world,ierr)
43. call mpi_recv(uold(1,bn-1),(n+1),mpi_real, &
my_rank-1,50,mpi_comm_world,status,ierr)
44. end if
45. if ((my_rank.gt.0).and.(my_rank.lt.p-1) &
.and.(mod(my_rank,2).eq.0)) then
46. call mpi_recv(uold(1,bn-1),(n+1),mpi_real, &
my_rank-1,50, mpi_comm_world,status,ierr)
47. call mpi_send(uold(1,bn),(n+1),mpi_real, &
my_rank-1,50, mpi_comm_world,ierr)
© 2004 by Chapman & Hall/CRC
272 CHAPTER 6. HIGH PERFORMANCE COMPUTING
Table 6.6.1: Processor Times for Diusion
p Times Speedups
2 87.2 1.0
4 41.2 2.1
8 21.5 4.1
16 11.1 7.9
32 06.3 13.8
48. call mpi_recv(uold(1,en+1),(n+1),mpi_real, &
my_rank+1,50, mpi_comm_world,status,ierr)
49. call mpi_send(uold(1,en),(n+1),mpi_real, &
my_rank+1,50, mpi_comm_world,ierr)
50. end if
51. if (my_rank.eq.p-1) then
52. call mpi_send(uold(1,bn),(n+1),mpi_real, &
my_rank-1,50, mpi_comm_world,ierr)
53. call mpi_recv(uold(1,bn-1),(n+1),mpi_real, &
my_rank-1,50, mpi_comm_world,status,ierr)
54. end if
55. end do
56. if (my_rank.eq.0) then
57. do source = 1,p-1
58. sbn = 2+(source)*loc_n
59. call mpi_recv(uold(1,sbn),(n+1)*loc_n,mpi_real, &
source,50, mpi_comm_world,status,ierr)
60. end do
61. else
62. call mpi_send(uold(1,bn),(n+1)*loc_n,mpi_real, &
0,50, mpi_comm_world,ierr)
63. end if
64. if (my_rank.eq.0) then
65. tend = timef()
66. print*, ’time =’, tend
67. print*, uold(2,2),uold(3,3),uold(4,4),uold(500,500)
68. end if
69. call mpi_finalize(ierr)
70. end
The co de can be compiled and executed on the IBM/SP by the following:
mpxlf90 —O4 heat2dmpi.f where mpxlf90 is the compiler, and using the load
leveler llsubmit envrmpi2 where envmpi2 contains the job parameters such as
time and number of processors. Table 6.6.1 contains the times in seconds to
execute the above file with di
erent numbers of processors s = 2> 4> 8> 16 and 32.
Good speedups relative to the execution time using two processors are recorded.
© 2004 by Chapman & Hall/CRC
6.6. MPI AND 2D MODELS 273
Table 6.6.2: Processor Times for Pollutant
p Times Speedups
2 62.9 1.0
4 28.9 2.2
8 15.0 4.2
16 07.9 8.0
32 04.6 13.7
6.6.3 Pollutant Transfer in Two Directions
A simple model for pollutant transfer in a shallow lake was formulated in Section
1.5.
Explicit Finite Di
erence 2D Pollutant Model: x
n
l>m
x(l{> m|> nw)=
x
n+1
l>m
= y
1
(w@{)x
n
l1>m
+ y
2
(w@|)x
n
l>m1
+ (6.6.4)
(1
y
1
(w@{) y
2
(w@|) w ghf)x
n
l>m
x
0
l>m
= given and (6.6.5)
x
n
0>m
and x
n
l>
0
= given= (6.6.6)
The MPI/Fortran code p ol l2dmpi.f is only a slight modification of the above
co de for heat di
usion. We have kept the same communication scheme. This is
not completely necessary because the wind is from the southwest, and therefore,
the new concentration will depend only on two of the adjacent space nodes, the
south and the west nodes. The initialization is similar, and the execution on
the processors for (6.6.4) is
do j = bn,en
do i= 2,n
unew(i,j) = dt*f + dt*velx/dx*uold(i-1,j)&
+ dt*vely/dy*uold(i,j-1) &
+ (1- dt*velx/dx - dt*vely/dy &
- dt*dec)*uold(i,j)
end do
end do.
The calculations that are recorded in Table 6.6.2 are for the number of proces-
sors
s = 2> 4> 8> 16 and 32 and have good speedups relative to the two processors
time.
6.6.4 Exercises
1. In heat2dmpi.f carefully study the communication scheme, and verify the
© 2004 by Chapman & Hall/CRC
communications for the case of eight processors as depicted in Figure 6.6.2.
274 CHAPTER 6. HIGH PERFORMANCE COMPUTING
2. In poll2dm.f study the communication scheme and delete any unused
mpi_send() and mpi_recv() subroutines. Also, try to use any of the mpi col-
lective subroutines such a mpi_gather().
3. In heat2dmpi.f and in poll2dmpi.f explain why the codes fail if only one
processor is used.
4. In poll2dm.f consider the case where the wind comes from the northwest.
Mo dify the discrete model and the code.
5.
erent n
and compare speedups.
6.
erent n
and compare speedups.
© 2004 by Chapman & Hall/CRC
Duplicate the computations in Table 6.6.1.Experiment with di
Duplicate the computations in Table 6.6.2.Experiment with di
Chapter 7
Message Passing Interface
In this chapter a more detailed discussion of MPI will be undertaken. The
basic eight MPI commands and the four collective communication subroutines
mpi_bcast(), mpi_reduce(), mpi_gather() and mpi_scatter() will be studied in
the first three sections. These twelve commands/subroutines form a basis for
all MPI programming, but there are many additional MPI subroutines. Section
7.4 describes three methods for grouping data so as to minimize the number of
calls to communication subroutines, which can have significant startup times.
Section 7.5 describes other possible communicators, which are just subsets of
the processors that are allowed to have communications. In the last section
these topics are applied to matrix-matrix products via Fox’s algorithm. Each
section has several short demonstration MPI codes, and these should be helpful
to the first time user of MPI. This chapter is a brief introduction to MPI, and
the reader should also consult other texts on MPI such as P. S. Pacheco [21]
and W. Gropp, E. Lusk, A. Skjellum and R. Thahur [8].
7.1 Basic MPI Subroutines
7.1.1 Intro duction
MPI programming can be done in either C or Fortran by using a library of
MPI subroutines. In the text we have used Fortran 9x and the MPI library is
called mpif.h. The text web site contains both the C and Fortran codes. The
following is the basic structure for MPI codes:
include ’mpif.h’
.
.
.
call mpi_init(ierr)
call mpi_comm_rank(mpi_comm_world, my_rank, ierr)
call mpi_comm_size(mpi_comm_world, p, ierr)
275
© 2004 by Chapman & Hall/CRC
In the last three sections in Chapter 6 several MPI codes were illustrated.
276 CHAPTER 7. MESSAGE PASSING INTERFACE
.
.
.
do parallel work
.
.
.
call mpi_barrier(mpi_comm_world, ierr)
.
.
.
do communications via mpi_send() and mpi_recv()
.
.
.
call mpi_finalize(ierr).
The parameters my_rank, ierr, and p are integers where p is the number
of pro cessors, which are listed from 0 to p-1. Each processor is identified by
my_rank ranging from 0 to p-1. Any error status is indicated by ierr. The para-
meter mpi_comm_world is a special MPI type, called a communicator, that is
used to identify subsets of processors having communication p atterns. Here the
generic communicator, mpi_comm_world, is the set of all p processors and all
processors are allowed to communicate with each other. Once the three calls to
mpi_init(), mpi_comm_rank() and mpi_comm_size() have been made, each
pro cessor will execute the code before the call to mpi_finalize(), which termi-
nates the parallel computations. Since each processor has a unique value for
my_rank, the code or the input data may be di
erent for each processor. The
call to mpi_barrier() is used to insure that each processor has completed its
computations. Any communications between processors may be done by calls
to mpi_send() and mpi _recv().
7.1.2 Syntax for mpi_send() and mpi_recv()
MPI has many dierent subroutines that can be used to do communications be-
tween processors. The communication subroutines mpi_send() and mpi_recv()
are the most elementary, and they must be called in pairs. That is, if processor
0 wants to send data to processor 1, then a mpi_send() from processor 0 must
be called as well as a mpi_recv() from pro cessor 1 must be called.
mpi_send(senddata, count, mpi_datatype, dest, tag, mpi_comm, ierr)
senddata array(*)
count integer
mpi_datatype integer
dest integer
tag integer
mpi_comm integer
ierr integer
There a number of mpi_datatypes, and some of these are mpi_real, mpi_int,
and mpi_char. The integer dest indicates the processor that data is to be sent.
The parameter tag is used to clear up any confusion concerning multiple calls
© 2004 by Chapman & Hall/CRC
7.1. BASIC MPI SUBROUTINES 277
to mpi_send(). The syntax for mpi_recv() is similar, but it does have one
additional parameter, mpi_status for the status of the communication.
mpi_rev(recvdata, count, mpi_datatyp e, source, tag
, mpi_comm, status, ierr)
recvdata array(*)
count integer
mpi_datatype integer
source integer
tag integer
mpi_comm integer
status(mpi_status_size) integer
ierr integer
Suppose processor 0 needs to communicate the real number, a, and the
integer, n, to the other p-1 processors. Then there must be 2(p-1) pairs of
calls to mpi_send() and mpi_recv(), and one must be sure to use di
erent tags
associated with a and n. The following if-else-endif will do this where the first
part of the if-else-endif is for only processor 0 and the second part has p-1 copies
with one for each of the other processors from 1 to p-1:
if (my_rank.eq.0) then
do dest = 1,p-1
taga = 0
call mpi_send(a, 1, mpi_real, dest, taga
, mpi_comm_world, ierr)
tagn = 1
call mpi_send(n, 1, mpi_int, dest, tagn
, mpi_comm_world, ierr)
end do
else
taga = 0
call mpi_recv(a, 1, mpi_real, 0, taga
, mpi_comm_world, status, ierr)
tagn = 1
call mpi_recv(n, 1, mpi_int, 0, tagn
, mpi_comm_world, status, ierr)
end if.
7.1.3 First MPI Code
This first MPI code simply partitions an interval from a to b into p equal
parts. The data in line 11 will be "hardwired" into all the processors because it
precedes the initialization of MPI in lines 14-15. Each processor will execute the
print commands in lines 17-19. Since my_rank will vary with each processor,
each processor will have unique values of loc_a and loc_b. The if_else_endif
© 2004 by Chapman & Hall/CRC
278 CHAPTER 7. MESSAGE PASSING INTERFACE
in lines 31-40 communicates all the loc_a to processor 0 and stores them in the
array a_list. The print commands in lines 26-28 and lines 43-47 verify this.
The outputs for the print commands might not appear in sequential order that
is indicated following the code listing. This output verifies the communications
for p = 4 processors.
MPI/Fortran 9x Code basicmpi.f
1. program basicmpi
2.! Illustrates the basic eight mpi commands.
3. implicit none
4.! Includes the mpi Fortran library.
5. include ’mpif.h’
6. real:: a,b,h,loc_a,loc_b,total
7. real, dimension(0:31):: a_list
8. integer:: my_rank,p,n,source,dest,tag,ierr,loc_n
9. integer:: i,status(mpi_status_size)
10.! Every processor gets values for a,b and n.
11. data a,b,n,dest,tag/0.0,100.0,1024,0,50/
12.! Initializes mpi, gets the rank of the processor, my_rank,
13.! and number of pro cessors, p.
14. call mpi_init(ierr)
15. call mpi_comm_rank(mpi_comm_world,my_rank,ierr)
16. call mpi_comm_size(mpi_comm_world,p,ierr)
17. print*,’my_rank =’,my_rank, ’a = ’,a
18. print*,’my_rank =’,my_rank, ’b = ’,b
19. print*,’my_rank =’,my_rank, ’n = ’,n
20. h = (b-a)/n
21.! Each processor has unique value of loc_n, loc_a and loc_b.
22. loc_n = n/p
23. loc_a = a+my_rank*loc_n*h
24. loc_b = loc_a + loc_n*h
25.! Each processor prints its loc_n, loc_a and loc_b.
26. print*,’my_rank =’,my_rank, ’loc_a = ’,lo c_a
27. print*,’my_rank =’,my_rank, ’loc_b = ’,loc_b
28. print*,’my_rank =’,my_rank, ’loc_n = ’,loc_n
29.! Processors p not equal 0 sends a_loc to an array, a_list,
30.! in processor 0, and processor 0 recieves these.
31. if (my_rank.eq.0) then
32. a_list(0) = loc_a
33. do source = 1,p-1
34. call mpi_recv(a_list(source),1,mpi_real,source &
35. ,50,mpi_comm_world,status,ierr)
36. end do
37. else
38. call mpi_send(loc_a,1,mpi_real,0,50,&
© 2004 by Chapman & Hall/CRC
7.1. BASIC MPI SUBROUTINES 279
39. mpi_comm_world,ierr)
40. end if
41. call mpi_barrier(mpi_comm_world,ierr)
42.! Pro cessor 0 prints the list of all loc_a.
43. if (my_rank.eq.0) then
44. do i = 0,p-1
45. print*, ’a_list(’,i,’) = ’,a_list(i)
46. end do
47. end if
48.! mpi is terminated.
49. call mpi_finalize(ierr)
50. end program basicmpi
my_rank = 0 a = 0.0000000000E+00
my_rank = 0 b = 100.0000000
my_rank = 0 n = 1024.
my_rank = 1 a = 0.0000000000E+00
my_rank = 1 b = 100.0000000
my_rank = 1 n = 1024
my_rank = 2 a = 0.0000000000E+00
my_rank = 2 b = 100.0000000
my_rank = 2 n = 1024
my_rank = 3 a = 0.0000000000E+00
my_rank = 3 b = 100.0000000
my_rank = 3 n = 1024
!
my_rank = 0 loc_a = 0.0000000000E+00
my_rank = 0 loc_b = 25.00000000
my_rank = 0 loc_n = 256
my_rank = 1 loc_a = 25.00000000
my_rank = 1 loc_b = 50.00000000
my_rank = 1 loc_n = 256
my_rank = 2 loc_a = 50.00000000
my_rank = 2 loc_b = 75.00000000
my_rank = 2 loc_n = 256
my_rank = 3 loc_a = 75.00000000
my_rank = 3 loc_b = 100.0000000
my_rank = 3 loc_n = 256
!
a_list( 0 ) = 0.0000000000E+00
a_list( 1 ) = 25.00000000
a_list( 2 ) = 50.00000000
a_list( 3 ) = 75.00000000
© 2004 by Chapman & Hall/CRC