Subroutines and exercises for the computer solution of problems involving matrices, integrals, differential equations, spline functions, zeros and extrema of functions, least squares, and Monte Carlo techniques. Computer Methods for ces IMaineMaliGal ComputationsCOMPUTER METHODS FOR MATHEMATICAL COMPUTATIONS GEORGE E. FORSYTHE MICHAEL A. MALCOLM Department of Computer Science University of Waterloo CLEVE B. MOLER Department of Mathematics and Statistics University of New Mexico PRENTICE-HALL, INC. ENGLEWOOD CLIFFS, N. 3. 07632Library of Consress Cataloging in Publication Data Forsrrie, Gronce, Eten, (date) ‘Computer methods for mathematical computations, (Prentice-Hall series in automatic computation) Sibiograpny: pe Includes index” 1. Numerical analysis —Data processing. 2. FORTRAN (Computer program language) 1. Malcolm, Michael A. (date) joint author. Il, Moler, Cleve B, joint author. IL. Title. QA207.F568 519.4 76-30819 ISBN 0-13-165332-6 © 1977 by Prentice-Hall, Inc., Englewood Cliffs, New Jersey 07632 All rights reserved. No part of this book may be reproduced in any form or by any means without permission in writing from the publisher. 098765 Printed in the United States of America PRENTICE-HALL INTERNATIONAL, INC., London PRENTICE-HALL OF AUSTRALIA PTY. LIMITED, Sydney PRENTICE-HALL OF CANADA, LTD., Toronto PRENTICE-HALL OF INDIA PRIVATE LIMITED, New Delhi PRENTICE-HALL OF JAPAN, INC., Tokyo PRENTICE-HALL OF SOUTHEAST ASIA PTE. LTD., Singapore WHITEHALL BOOKS LiMiTED, Wellington, New ZealandCONTENTS PREFACE INTRODUCTION 1.1 Bibliography 2 1.2 About the programs in this book 7 Problems & FLOATING-POINT COMPUTATION 2.1 Floating-point numbers 10 2.2 Calculation of machine epsilon 13 2.3 An example of round-off error 14 2.4 Instability of certain algorithms 16 2.5 Sensitivity of certain problems 17 2.6 Solving quadratic equations 20 Problems 23 LINEAR SYSTEMS OF EQUATIONS 3.1 Linear systems for stored matrices 32 3.2 Condition of a matrix 41 3.3 Subroutines DECOMP and SOLVE 48 3.4 Large, sparse systems 56 Problems 58 vii 10viii 4 CONTENTS INTERPOLATION 63 4.1 Polynomial interpolation 64 4.2 Evaluation of polynomials 68 4.3 An example, Runge’s function 69 4.4 Spline interpolation 70 4.5 Subroutines SPLINE and SEVAL 76 Problems 80 NUMERICAL INTEGRATION 84 5.1 The rectangle and trapezoid rules 85 5.2 Spline quadrature 89 5.3 Simpson's rule 91 5.4 Adaptive quadrature routines 92 5.5 Subroutine QUANC8 97 Problems 106 INITIAL VALUE PROBLEMS IN ORDINARY DIFFERENTIAL EQUATIONS 110 6.1 The problem to be solved 1/0 6.2 Numerical solutions 112 6.3 Errors 114 64 Methods 1/9 6.5 Stiff equations 123 6.6 Boundary value problems 126 6.7 Choice of a subroutine 127 6.8 Subroutine RKF45 129 Problems 148 SOLUTION OF NONLINEAR EQUATIONS 156 7.1 Transcendental equations—real roots 156 7.2 Subroutine ZEROIN 16/ 7.3 Transcendental equations—complex roots 167 7.4 Zeros of polynomials 168 7.5 Nonlinear systems of equations 169 Problems 17/10 OPTIMIZATION 8.1 One-dimensional optimization 179 8.2 Subroutine FMIN 182 8.3 Optimization in many dimensions 188 Problems 190 LEAST SQUARES AND THE SINGULAR VALUE DECOMPOSITION 9.1 Least squares data fitting 192 9.2 Orthogonality and the SVD 20/ 9.3 Applications 207 9.4 Computing the decomposition 2/8 9.5 Subroutine SVD 227 Problems 236 RANDOM NUMBER GENERATION AND MONTE CARLO METHODS 10.1 Generation of uniformly distributed numbers 241 10.2 Subroutine URAND 245 10.3 Sampling from other distributions 247 Problems 248 REFERENCES INDEX CONTENTS ix 178 192 240 250 255There are, so to speak, in the mathematical country, precipices and pit-shafts down which it would be possible to fall, but that need not deter us from walking about. L. F. Richardson PREFACE This book evolved from a set of notes used in introductory numerical methods courses at Stanford University, the University of New Mexico and the University of Waterloo. These universities have two such courses, titled something like “Numerical Computing” and “Numerical Analysis”. The numerical computing course is for students of science and engineering who simply want to use numerical methods in their work. It lasts only one term and has minimal mathematical prerequisites. The numerical analysis course covers essentially the same methods, but lasts a full year. The additional time is devoted to deeper mathematical analysis. This book is intended primarily for the numerical computing course, but could also be used to supplement a more theoretical text in the numerical analysis course. Ten Fortran subroutines are included in the book. It is expected that they will be used extensively both during and after the course. The programs have been carefully written so that they may be easily used on a wide variety of computers with little or no modification. One of our colleagues has pointed out that “it is an order of magnitude easier to write two good subroutines than it is to decide which one is best”. In choosing among the various subroutines available for a particular prob- lem, we have placed considerable emphasis on the clarity and style of pro- gramming. If several subroutines have comparable accuracy, reliability and efficiency, we have chosen the one which is the least difficult to read and use. In recent years, there has been increased recognition of the importance of writing, testing and distributing quality mathematical software. So, although we believe that our subroutines are among the best available at the time the book is being written, we hope and expect that they will eventually be superceded. One of our goals in writing the book is to help the reader xPREFACE — Xi look for better subroutines and recognize and appreciate them when they become available. A copy of the subroutines in 1966 ANSI Standard Fortran is available from the authors. Both single and double precision versions are provided on a single tape. The tape is provided in two formats: 7-track, BCD, 556 BPI, and 9-track, EBCDIC, 800 BPI. A deck containing about 2000 cards is also available upon special request. Write to: Prof. Cleve Moler, Depart- ment of Mathematics, University of New Mexico, Albuquerque, N.M. 87131. We are indebted to many people for help and encouragement during the preparation of the manuscript. Following is a list of some of those people: Richard Allen, Mary Bodley, John Bolstad, Erin Brent, Richard Brent, Francena Brumbaugh, Andy Conn, Carl deBoor, Fred Dorr, C. William Gear, Keith Geddes, Alan George, Gene Golub, Dennis Jerpersen, Rondall Jones, William Kahan, David Kahaner, Shaaron Kent, Fred Krogh, Doug Lawson, James Lyness, Kathryn Moler, Teresa Moler, Walter Murray, Larry Nazareth, Joe Oliger, Victor Pereyra, Steve Pruess, George Ramos, Laurie Rogers, Michael Saunders, Larry Shampine, Laura Staggers, G. W. Stewart, Mike Steuerwalt, David Stoutemyer, Richard Underwood, H. A. Watts, and Michael Wester. Thanks to Masatake Mori and Robert Steel who pointed out errors in the first printing. Special thanks are due to the many students who suffered through early versions of the notes and helped debug the sub-routines as well. Prior to the completion of this manuscript, on April 9, 1972, Professor George E. Forsythe met an untimely death at the age of 55. At the time, Professor Forsythe was the chairman and founder of Stanford's Computer Science Department. His loss was mourned by the academic and scientific communities, while we, George’s friends, colleagues, and students, felt a very personal loss. In his tribute given at the memorial service, Professor Edward A. Feigenbaum spoke for many of us when he said of Professor Forsythe: “He was one of the gentlest and most humane of people, and we loved him for it... He was a man without violence of the spirit . . .” The spirit of this book is George Forsythe’s. It is our desire that it find peaceful uses worthy of his memory. Micuaet A. MALCOLM University of Waterloo Cueve B. MOLER University of New Mexico1 INTRODUCTION This book is concerned with solving mathematical problems using auto- matic digital computers. The reader is assumed to have completed two years of university mathematics. This should include differential and integral cal- culus and a little matrix theory and differential equations. He is also assumed to have access to a medium- or large-scale scientific computer and be able to program it using the local dialect of Fortran. A very important part of the book is a set of Fortran subroutines. In fact, the book might well be regarded as an extensive “user’s guide” for the sub- routines. These are not the simple, illustrative sample programs found in many textbooks but rather are representative of the state of the art in current scientific computing. We intend that the Fortran subroutines be read by human readers as well as machines; that is why they are printed in the text. We also intend that the programs be used to solve problems given in the text. We have followed a rather unconventional philosophy with regard to the presentation of material. Most engineers and scientists have neither the time nor the inclination to keep abreast of current literature in numerical analysis. We have observed that many people solving practical numerical problems rely upon the methods, if not the programs, used in previous course work. There- fore, we decided to present up-to-date subroutines for solving standard problems of mathematics. Many of these subroutines are very intricate, and to describe in detail how and why they work would require much longer than most students can spend on this type of material. Hence, it is necessary to treat many of these sub- routines as black boxes. Treating subroutines as black boxes is a common practice. The hardware subroutines for such things as addition and mul- tiplication are treated as black boxes by most computer programmers. Few people really understand how they work. For the past several years, sub- 12 INTRODUCTION cHap. 1 routines used to compute standard functions of analysis (e.g., sin, cos, etc.) have been treated as black boxes, and few programmers understand them in any detail. Such subroutines are accepted as primitives which are expected to work. It is the opinion of the authors that it is now time for programmers to accept subroutines which solve linear systems, ordinary differential equa- tions, and many other standard mathematical problems as primitives which (usually) work. However, it is not sufficient to simply explain how to call each of the sub- routines and say that they “usually work.” There is a wealth of pitfalls in numerical computation. The student must be properly warned of these pit- falls. He should learn to look for symptoms of numerical ill health and to correctly diagnose the problem(s). This requires a certain level of understand- ing of the numerical methods employed by the various subroutines. The programmer will also find occasion to modify a subroutine or use a variation of a method to solve a related problem. This usually requires more detailed knowledge. We have tried to compromise on the amount of detail included with each method. The student may come to regard the subroutines as “gray boxes” rather than black. Where appropriate, we have referenced the literature for further discussion of a particular topic. A short discussion of various decisions and style considerations that went into the Fortran subroutines is included in Section 1.2 of this chapter. Since we shall be able to cover only a small part of the subject, we shall rely heavily on references to other books and articles, So, we begin our pres- entation with the bibliography. 1.1, BIBLIOGRAPHY Two introductory books, mainly intended for the desk-machine user but having a sub- stantial carry-over to automatic computing: Hipesranp, F. B. (1974), Introduction to numerical analysis, 2nd ed. New York: McGraw-Hill. Lanczos, C. (1957), Applied analysis. Englewood Cliffs, N.J.: Prentice-Hall. Some general books, in which automatic computers are considered: Acton, F. S. (1970), Numerical methods that (usually) work. New York: Harper & Row. (Two especially good chapters on removing singularities.) ‘Ames, W. F. (1969), Numerical methods for partial differential equations. New York: Barnes & Noble. Berezin, I.S., and N. P. ZHiKov (1965), Computing methods, 2 vols. Elmsford, N.Y.: Pergamon. (A translation by Blunn of the 1959 textbook at Moscow State University for a two-year course.)sec. 1.1 BIBLIOGRAPHY = 3 Conte, S. D., and C. DeBoor (1972), Elementary numerical analysis; an algorithmic approach, 2nd ed. New York: McGraw-Hill. Dantourst, G., and A. Bsérck (1974), Numerical methods, Englewood Cliffs, N.J.: Prentice-Hall. (Complete, up-to-date text in numerical analysis.) Fox, L., and D. F. Mayers (1968), Computing methods for scientists and engineers. Oxford: Clarendon Press. HamMine, R. W. (1971), Introduction to applied numerical analysis. New York: McGraw-Hill. Hammino, R. W. (1973), Numerical methods for scientists and engineers, 2nd ed. New York: McGraw-Hill. (Hamming wants you to combine mathematical theory, heuristical analysis, and computer methods in the solution of your problem: “The purpose of computing is insight, not numbers.”) Henrict, P. (1964), Elements of numerical analysis. New York: Wiley. (Elementary fundamentals, done very carefully. Nothing on matrix problems.) McCracken, D. D., and W. S. Dorn (1964), Numerical methods and Fortran pro- gramming. New York: Wiley. RAtston, A. (1965), A first course in numerical analysis. New York: McGraw-Hill. (In contrast to Henrici and Hamming, Ralston has a substantial amount on computational methods of linear algebra.) Ratston, A., and H. S. WILF (1960, 1967), Mathematical methods for digital com- puters, vol. 1 and vol. 2. New York: Wiley. (Mathematical descriptions and flowcharts of algorithms of varying quality. Those in Volume 2 are generally much better.) Suamprne, L., and R. ALLEN (1973), Numerical computing. Philadelphia: Saunders. (A text similar in spirit to this one, Contains several good Fortran subroutines.) Topp, J. (editor) (1962), Survey of numerical analysis. New York: McGraw-Hill. WenprorF, B. (1969), First principles of numerical analysis. An undergraduate text. Reading, Mass.: Addison-Wesley. Bibliography of tables: Fuercuer, A., J.C. P. MILLER, and L. RosENHEAD (1962), Index of mathematical tables, 2nd ed., 2 vols. Oxford: Blackwell’s. (This lists nearly every published table of mathematical functions up to around 1958 to 1959, and there were many. Indispensable.) Lists of elementary functions that approximate transcendental functions: Hart, J. F., et al. (1968), Computer approximations. New York: Wiley. Hastinos, C., Jr., et al. (1955), Approximations for digital computers. Princeton, N.J.: Princeton University Press. (Formulas with graphs of error curves. Most computer library routines are based on these formulas or improvements thereof.)