TERM PAPER CSE316

MEMORY MANAGEMENT IN LINUX & UNIX

Submitted to: MR.MANISH KUMAR

Submitted by: SURENDRA MCA 2nd SEM *roll no D3804B51 *Regd. no 10806601

ACKNOWLEDGMENT
I have the honor to express our gratitude to our Prof. Mr. Manish Kumar for their timely help, support encouragement, valuable comments, suggestions and many innovative ideas in carrying out this project. It is great for me to gain out of their experience. I also owe overwhelming debt to my classmates for their constant encouragement and support throughout the execution of this project. Above all, consider it a blessing of the Almighty God that i have been able to complete this project. His contribution towards every aspect of ours throughout our life would always be the supreme and unmatched.

SURENDRA SINGH

TABLE OF CONTENTS
UNIX :INTRODUCTION.

UNIX

INTRODUCTION

Unix is popular operating system, developed by AT&T in 1969 and it has been very important to the development of the Internet. It is a multi-processing, multi-user, family of operating systems that run on a variety of architechtures. UNIX allows more than one user to access a computer system at the same time. A widely used Open Source Unix-like operating system kernel. Linux was first released by its inventor Linus Torvalds in 1991. Combining the Linux kernel with the GNU software forms the basis of the operating system family generally known as 'Linux'. There are distributions of GNU/Linux for almost every available type of computer hardware from desktop machines to IBM mainframes. The inner workings of GNU/Linux are open and available for anyone to examine and change as long as they make their changes available to the public, as set out in the terms of the GNU General Public License. Because of its robustness and availability, Linux has won popularity in the Open Source community as well as among commercial application developers. Here is more input:

Unix requires a more powerful hardware configuration. It will work in large mainframe computers but will not work in an x86 based personal computer. Linux however, (which is built on the concept of Unix) has small hardware requirements and it will work on both a large mainframe computer and an x86 based personal computer. Unix is an Operating System developed in olden days in which the kernel, the heart of the OS, interacts directly with the hardware. (note: this is the definition of what a kernel is). Because UNIX treats everything as a file, it provides greater security for users. An

example of a UNIX distribution is posix. (note: actually POSIX is a set of standards for interoperability of applications between UNIX and UNIX-like systems). Linux uses a the UNIX architecture as its basis and provides more facilities and applications. Linux could be considered to be a GUI to the UNIX core. (note: this is plain wrong. GNU/ Linux was rewritten from scratch using UNIX as a guide. GNOME and KDE are GUIs for GNU/Linux). Examples of Linux distributions are Redhat, Fedora, Susee, Mandriva, and Ubuntu. Solaris OS also uses the UNIX kernal almost all UNIX commands will work on solaris in addition to 500 Solaris specific commands. (note: Solaris is also a rewrite of UNIX for x86, and does not use any original UNIX code). Both UNIX and LINUX are Open source. (note: UNIX is proprietary, Linux is open source)

Unix is the foundation for a number of operating systems, with Linux being the most popular one. Novell and Free BSD are 2 other commonly used Unix varients.(note: Again, the BSD family are based on another rewrite of UNIX for x86, UNIX is not their foundation in the sense implied here) UNIX is an operating system created in the early days of computers. More recently, Linux was created as an open-source, freeware operating system. (note: Linux is free software, not freeware. Free software is open source that insists any developer reusing code releases their own work as free software. Freeware is proprietary software distributed at no cost [gratis]) It is "UNIX-LIKE", meaning that it uses many UNIX constructs but also departs from traditional UNIX in many ways. Like UNIX, Linux is faster than many of the other commercially available operating systems. (note: This is a sweeping generalization and depends on the hardware used, and what servers and applications are running) It appears to also be far more robust than any of the Microsoft products. Linux is being used in many time critical applications because of it's speed. It is also used in many applications that need to maintain uptime because Linux, like UNIX, can run for months at a time without rebooting. While the typical method of solving Microsoft problems is to "reboot", that particular requirement does not seem to be appropriate in a Linux/Unix environment. While UNIX has created a windows-like work environment, Linux has improved greatly on that concept. Linux has become a real player in the consumer operating system market... and it's free. While you may want to pay for a Linux distribution, the actual code is free and you are allowed to load it on as many machines as you want. You can get Linux for free if you wish to load it across the internet.

HISTORY

Since it began to escape from AT&T's Bell Laboratories in the early 1970's, the success of the UNIX operating system has led to many different versions: recipients of the (at that time free) UNIX system code all began developing their own different versions in their own, different, ways for use and sale. Universities, research institutes, government bodies and computer companies all began using the powerful UNIX system to develop many of the technologies which today are part of a UNIX system.

Computer aided design, manufacturing control systems, laboratory simulations, even the Internet itself, all began life with and because of UNIX systems. Today, without UNIX systems, the Internet would come to a screeching halt. Most telephone calls could not be made, electronic commerce would grind to a halt and there would have never been "Jurassic Park"! By the late 1970's, a ripple effect had come into play. By now the under- and post-graduate students whose lab work had pioneered these new applications of technology were attaining management and decision-making positions inside the computer system suppliers and among its customers. And they wanted to continue using UNIX systems. Soon all the large vendors, and many smaller ones, were marketing their own, diverging, versions of the UNIX system optimized for their own computer architectures and boasting many different strengths and features. Customers found that, although UNIX systems were available everywhere, they seldom were able to interwork or co-exist without significant investment of time and effort to make them work effectively. The trade mark UNIX was ubiquitous, but it was applied to a multitude of different, incompatible products. In the early 1980's, the market for UNIX systems had grown enough to be noticed by industry analysts and researchers. Now the question was no longer "What is a UNIX system?" but "Is a UNIX system suitable for business and commerce?" Throughout the early and mid-1980's, the debate about the strengths and weaknesses of UNIX systems raged, often fuelled by the utterances of the vendors themselves who sought to protect their profitable proprietary system sales by talking UNIX systems down. And, in an effort to further differentiate their competing UNIX system products, they kept developing and adding features of their own. In 1984, another factor brought added attention to UNIX systems. A group of vendors concerned about the continuing encroachment into their markets and control of system interfaces by the larger companies, developed the concept of "open systems." Open systems were those that would meet agreed specifications or standards. This resulted in the formation of X/Open Company Ltd whose remit was, and today in the guise of The Open Group remains, to define a comprehensive open systems environment. Open systems, they declared, would save on costs, attract a wider portfolio of applications and competition on equal terms. X/Open chose the UNIX system as the platform for the basis of open systems. Although UNIX was still owned by AT&T, the company did little commercially with it until the mid1980's. Then the spotlight of X/Open showed clearly that a single, standard version of the UNIX system would be in the wider interests of the industry and its customers. The question now was, "which version?". In a move intended to unify the market in 1987, AT&T announced a pact with Sun Microsystems, the leading proponent of the Berkeley derived strain of UNIX. However, the rest of the industry viewed the development with considerable concern. Believing that their own markets were under threat they clubbed together to develop their own "new" open systems operating system. Their new organization was called the Open Software Foundation (OSF). In response to this, the AT&T/Sun faction formed UNIX International. The ensuing "UNIX wars" divided the system vendors between these two camps clustered around the two dominant UNIX system technologies: AT&T's System V and the OSF system called OSF/1. In the meantime, X/Open Company held the center ground. It continued the process of standardizing the APIs necessary for an open operating system specification. In addition, it looked at areas of the system beyond the operating system level where a standard approach would add value for supplier and customer alike, developing or adopting specifications for

languages, database connectivity, networking and mainframe interworking. The results of this work were published in successive X/Open Portability Guides. XPG 4 was released in October 1992. During this time, X/Open had put in place a brand program based on vendor guarantees and supported by testing. Since the publication of XPG4, X/Open has continued to broaden the scope of open systems specifications in line with market requirements. As the benefits of the X/Open brand became known and understood, many large organizations began using X/Open as the basis for system design and procurement. By 1993, over $7 billion had been spent on X/Open branded systems. By the start of 1997 that figure has risen to over $23 billion. To date, procurements referencing the Single UNIX Specification amount to over $5.2 billion. In early 1993, AT&T sold it UNIX System Laboratories to Novell which was looking for a heavyweight operating system to link to its NetWare product range. At the same time, the company recognized that vesting control of the definition (specification) and trademark with a vendor-neutral organization would further facilitate the value of UNIX as a foundation of open systems. So the constituent parts of the UNIX System, previously owned by a single entity are now quite separate In 1995 SCO bought the UNIX Systems business from Novell, and UNIX system source code and technology continues to be developed by SCO. In 1995 X/Open introduced the UNIX 95 brand for computer systems guaranteed to meet the Single UNIX Specification. The Single UNIX Specification brand program has now achieved critical mass: vendors whose products have met the demanding criteria now account for the majority of UNIX systems by value. For over ten years, since the inception of X/Open, UNIX had been closely linked with open systems. X/Open, now part of The Open Group, continues to develop and evolve the Single UNIX Specification and associated brand program on behalf of the IT community. The freeing of the specification of the interfaces from the technology is allowing many systems to support the UNIX philosophy of small, often simple tools , that can be combined in many ways to perform often complex tasks. The stability of the core interfaces preserves existing investment, and is allowing development of a rich set of software tools. The Open Source movement is building on this stable foundation and is creating a resurgence of enthusiasm for the UNIX philosophy. In many ways Open Source can be seen as the true delivery of Open Systems that will ensure it continues to go from strength to strength. 1969 The Beginning The history of UNIX starts back in 1969, when Ken Thompson, Dennis Ritchie and others started working on the "little-used PDP-7 in a corner" at Bell Labs and what was to become UNIX. It had a assembler for a PDP-11/20, file system, fork(), roff and ed. It was used for text processing of patent documents. It was rewritten in C. This made it portable and changed the history of OS's. UNIX leaves home. Also widely known as Version 6, this is the first to be widely available out side of Bell Labs. The first BSD version (1.x) was derived from V6. It was a "improvement over all preceding and following Unices" [Bourne]. It had C, UUCP and the Bourne shell. It was ported to the VAX and the kernel was more than 40 Kilobytes (K). Microsoft introduces Xenix. 32V and 4BSD introduced. AT&T's UNIX System Group (USG) release System III, the first public release outside Bell Laboratories. SunOS 1.0 ships. HP-UX introduced. Ultrix-11 Introduced. Computer Research Group (CRG), UNIX System Group (USG) and a third group merge to become UNIX System Development Lab. AT&T announces UNIX System V, the first supported release. Installed base 45,000. University of California at Berkeley releases 4.2BSD, includes TCP/IP, new

1971 1973 1975

First Edition Fourth Edition Sixth Edition

1979

Seventh Edition

1980 1982

Xenix System III

1983

System V

1984

4.2BSD

1984 1986 1987 1988 1989 1989 1990 1991

SVR2 4.3BSD SVR3

SVR4 XPG3

1992

SVR4.2

1993 Late 1993

4.4BSD SVR4.2MP

1994

Single UNIX Specification

1995

UNIX 95

1996 1997

Single UNIX Specification, Version 2 UNIX 98

1998

1999

UNIX at 30

2001

Single UNIX Specification, Version 3 ISO/IEC 9945:2003

2003

signals and much more. X/Open formed. System V Release 2 introduced. At this time there are 100,000 UNIX installations around the world. 4.3BSD released, including internet name server. SVID introduced. NFS shipped. AIX announced. Installed base 250,000. System V Release 3 including STREAMS, TLI, RFS. At this time there are 750,000 UNIX installations around the world. IRIX introduced. POSIX.1 published. Open Software Foundation (OSF) and UNIX International (UI) formed. Ultrix 4.2 ships. AT&T UNIX Software Operation formed in preparation for spinoff of USL. Motif 1.0 ships. UNIX System V Release 4 ships, unifying System V, BSD and Xenix. Installed base 1.2 million. X/Open launches XPG3 Brand. OSF/1 debuts. Plan 9 from Bell Labs ships. UNIX System Laboratories (USL) becomes a company - majority-owned by AT&T. Linus Torvalds commences Linux development. Solaris 1.0 debuts. USL releases UNIX System V Release 4.2 (Destiny). October - XPG4 Brand launched by X/Open. December 22nd Novell announces intent to acquire USL. Solaris 2.0 ships. 4.4BSD the final release from Berkeley. June 16 Novell acquires USL Novell transfers rights to the "UNIX" trademark and the Single UNIX Specification to X/Open. COSE initiative delivers "Spec 1170" to X/Open for fasttrack. In December Novell ships SVR4.2MP , the final USL OEM release of System V BSD 4.4-Lite eliminated all code claimed to infringe on USL/Novell. As the new owner of the UNIX trademark, X/Open introduces the Single UNIX Specification (formerly Spec 1170), separating the UNIX trademark from any actual code stream. X/Open introduces the UNIX 95 branding programme for implementations of the Single UNIX Specification. Novell sells UnixWare business line to SCO. Digital UNIX introduced. UnixWare 2.0 ships. OpenServer 5.0 debuts. The Open Group forms as a merger of OSF and X/Open. The Open Group introduces Version 2 of the Single UNIX Specification, including support for realtime, threads and 64-bit and larger processors. The specification is made freely available on the web. IRIX 6.4, AIX 4.3 and HP-UX 11 ship. The Open Group introduces the UNIX 98 family of brands, including Base, Workstation and Server. First UNIX 98 registered products shipped by Sun, IBM and NCR. The Open Source movement starts to take off with announcements from Netscape and IBM. UnixWare 7 and IRIX 6.5 ship. The UNIX system reaches its 30th anniversary. Linux 2.2 kernel released. The Open Group and the IEEE commence joint development of a revision to POSIX and the Single UNIX Specification. First LinuxWorld conferences. Dot com fever on the stock markets. Tru64 UNIX ships. Version 3 of the Single UNIX Specification unites IEEE POSIX, The Open Group and the industry efforts. Linux 2.4 kernel released. IT stocks face a hard time at the markets. The value of procurements for the UNIX brand exceeds $25 billion. AIX 5L ships. The core volumes of Version 3 of the Single UNIX Specification are approved as an international standard. The "Westwood" test suite ship for the UNIX 03 brand. Solaris 9.0 E ships. Linux 2.6 kernel released.

UNIX: Memory Management

Memory Management is the key behind the idea of sharing. In order to increase performance and have several processes in memory at a time, you have to find a way to share the memory.

This Memory consists of large array of words or bytes, each having its own address.

UNIX uses two methods for Memory Management:

Swapping

Paging What is swapping? What is paging?

Method 1: Swapping Memory

Processes need to be in memory to be executed. These processes can be put temporarily out of memory then brought back into memory for continued execution. (Swap)

Allocation of Main Memory and Swap Memory is done first. If the size of a process increases it does not fragment it relocates itself as a whole image into another part of memory.
1

If there is no piece of memory large enough then the process is swapped out till it can be swapped back in with a new size. Scheduler Process decides what processes are swapped in and out. This swapper will wake up once every 4 seconds to check the processes going in and out. If the process is idle, been in main memory too long, or is too large - it will most likely be swapped out. If a process is small or been swapped out for a long time it will most likely be swapped in. If processes do not need to be swapped at all the process table will search for worthwhile processes to take its place.

Many UNIX systems use this kind of swapping method. However, Berkeley designed systems depend on Paging for memory management.

Method 2: Paging
Paging allows for non contiguous address spacing. Phys Memory is broken into fixed blocks: Frames Log Memory is broken into fixed blocks: Pages When a process executes, the pages are loaded into any available memory frames from the backing store.

This backing store then is divided into the fixed blocks so that they are the same size as the memory frames.

A Page table exists between Logical and Physical memory

which contains base addresses for each page in the physical memory.

Berkeley Paging Methods: Berkeley introduced something called demand-paged-virtualmemory system, which eliminated external fragmentation.

Swapping was kept minimal because more than one job could be kept in main memory. Paging allows execution with only parts of a process in memory.

Demand Paging: 1. When a process needs a page and the page is not present - a page fault to the kernel occurs.
2. 3.

1.

2. A frame of memory is then allocated. 3. The proper disk page is read into a frame.

Interaction and Thrashing: CPU scheduling, memory swapping, and paging all interact together. If the priority is lower in a process, it is more likely that its paged will be paged/swapped entirely out. Having priority preferences like this guards against overworking the CPU. (Thrashing).

LINUX: INTRODUCTION
The creation of Linux

Linus Torvalds in 2002

In 1991, in Helsinki, Linus Torvalds began a project that later became the Linux kernel. It was initially a terminal emulator, which Torvalds used to access the large UNIX servers of the university. He wrote the program specifically for the hardware he was using and independent of an operating system because he wanted to use the functions of his new PC with an 80386 processor. Development was done on Minix using the GNU C compiler, which is still the main choice for compiling Linux today (although the code can be built with other compilers, such as the Intel C Compiler).[citation needed] As Torvalds wrote in his book Just for Fun[6], he eventually realized that he had written an operating system kernel. On 25 August 1991, he announced this system in a Usenet posting to the newsgroup "comp.os.minix.":[7]

Hello everybody out there using minix I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and I'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-) Linus (torvalds@kruuna.helsinki.fi) PS. Yes it's free of any minix code, and it has a multi-threaded fs. It is NOT portable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.

Controversy over Linux

Linux has been surrounded by controversy repeatedly since its inception.

"Linux is obsolete"

In 1992 Andrew S. Tanenbaum, recognized computer scientist and author of the Minix microkernel system, wrote a Usenet article on the newsgroup comp.os.minix with the title "Linux is obsolete,"[16] which marked the beginning of a famous debate about the structure of the then-recent Linux kernel. Among the most significant criticisms were that:

The kernel was monolithic and thus old-fashioned. The lack of portability, due to the use of exclusive features of the Intel 386 processor. "Writing a new operating system that is closely tied to any particular piece of hardware, especially a weird one like the Intel line, is basically wrong."[ There wasn't strict control of the source code by any individual person. Linux employed a set of features which were useless (Tanenbaum believed that multithreaded file systems were simply a "performance hack").

Tanenbaum's prediction that Linux would become outdated within a few years and replaced by GNU Hurd (which he considered to be more modern) proved incorrect. Linux has been ported to all major platforms and its open development model has led to an exemplary pace of development. In contrast, GNU Hurd has not yet reached the level of stability that would allow it to be used on a production server.

Linux Memory Management

Linux uses segmentation + pagination, which simplifies notation. Segments Linux uses only 4 segments:

2 segments (code and data/stack) for KERNEL SPACE from [0xC000 0000] (3 GB) to [0xFFFF FFFF] (4 GB) 2 segments (code and data/stack) for USER SPACE from [0] (0 GB) to [0xBFFF FFFF] (3 GB)

__ 4 GB--->| | | | Kernel | | Kernel Space (Code + Data/Stack) | | __| 3 GB--->|----------------| __ | | | | | | 2 GB--->| | | | Tasks | | User Space (Code + Data/Stack) | | | 1 GB--->| | | | | | |________________| __| 0x00000000 Kernel/User Linear addresses

7.2 Specific i386 implementation Again, Linux implements Pagination using 3 Levels of Paging, but in i386 architecture only 2 of them are really used: -----------------------------------------------------------------L I N E A R A D D R E S S -----------------------------------------------------------------\___/ \___/ \_____/ PD offset PF offset Frame offset [10 bits] [10 bits] [12 bits] | | | | | ----------| | | | Value |----------|--------| | | | |---------| /|\ | | | | | | | | | | | | | | | | | | Frame offset | | | | | | | \|/ | | | | | |---------|<-----| | | | | | | | | | | | | | | | x 4096 | | | | PF offset|_________|------| | | | /|\ | | | PD offset |_________|----- | | | _________| /|\ | | | | | | | | | | | \|/ | | \|/ _____ | | | ------>|_________| PHYSICAL ADDRESS | | \|/ | | x 4096 | | | CR3 |-------->| | | | |_____| | ....... | | ....... | | | | | Page Directory Page File

Linux i386 Paging

Linux Page Tables

Figure 3.3: Three Level Page Tables

7.3 Memory Mapping Linux manages Access Control with Pagination only, so different Tasks will have the same segment addresses, but different CR3 (register used to store Directory Page Address), pointing to different Page Entries. In User mode a task cannot overcome 3 GB limit (0 x C0 00 00 00), so only the first 768 page directory entries are meaningful (768*4MB = 3GB). When a Task goes in Kernel Mode (by System call or by IRQ) the other 256 pages directory entries become important, and they point to the same page files as all other Tasks (which are the same as the Kernel). Note that Kernel (and only kernel) Linear Space is equal to Kernel Physical Space, so: ________________ _____ |Other KernelData|___ | | | |----------------| | |__| | | Kernel |\ |____| Real Other | 3 GB --->|----------------| \ | Kernel Data | | |\ \ | | | __|_\_\____|__ Real | | Tasks | \ \ | Tasks | | __|___\_\__|__ Space | | | \\| | | | \ \|----------------| | | \ |Real KernelSpace|

|________________| Logical Addresses

\|________________| Physical Addresses

Linear Kernel Space corresponds to Physical Kernel Space translated 3 GB down (in fact page tables are something like { "00000000", "00000001" }, so they operate no virtualization, they only report physical addresses they take from linear ones). Notice that you'll not have an "addresses conflict" between Kernel and User spaces because we can manage physical addresses with Page Tables. 7.4 Low level memory allocation Boot Initialization We start from kmem_cache_init (launched by start_kernel [init/main.c] at boot up). |kmem_cache_init |kmem_cache_estimate kmem_cache_init [mm/slab.c] kmem_cache_estimate Now we continue with mem_init (also launched by start_kernel[init/main.c]) |mem_init |free_all_bootmem |free_all_bootmem_core mem_init [arch/i386/mm/init.c]

free_all_bootmem [mm/bootmem.c] free_all_bootmem_core Run-time allocation Under Linux, when we want to allocate memory, for example during "copy_on_write" mechanism (see Cap.10), we call: |copy_mm |allocate_mm = kmem_cache_alloc |__kmem_cache_alloc |kmem_cache_alloc_one |alloc_new_slab |kmem_cache_grow |kmem_getpages |__get_free_pages |alloc_pages |alloc_pages_pgdat |__alloc_pages |rmqueue |reclaim_pages Functions can be found under:

copy_mm [kernel/fork.c] allocate_mm [kernel/fork.c] kmem_cache_alloc [mm/slab.c] __kmem_cache_alloc kmem_cache_alloc_one alloc_new_slab kmem_cache_grow kmem_getpages __get_free_pages [mm/page_alloc.c] alloc_pages [mm/numa.c]

alloc_pages_pgdat __alloc_pages [mm/page_alloc.c] rm_queue reclaim_pages [mm/vmscan.c]

TODO: Understand Zones 7.5 Swap Overview Swap is managed by the kswapd daemon (kernel thread). kswapd As other kernel threads, kswapd has a main loop that wait to wake up. |kswapd |// initialization routines |for (;;) { // Main loop |do_try_to_free_pages |recalculate_vm_stats |refill_inactive_scan |run_task_queue |interruptible_sleep_on_timeout // we sleep for a new swap request |}

kswapd [mm/vmscan.c] do_try_to_free_pages recalculate_vm_stats [mm/swap.c] refill_inactive_scan [mm/vmswap.c] run_task_queue [kernel/softirq.c] interruptible_sleep_on_timeout [kernel/sched.c]

The Linux Page Cache

Figure 3.6: The Linux Page Cache

When do we need swapping?

Swapping is needed when we have to access a page that is not in physical memory. Linux uses ''kswapd'' kernel thread to carry out this purpose. When the Task receives a page fault exception we do the following: | Page Fault Exception | cause by all these conditions:
1

| a-) User page | b-) Read or write access | c-) Page not present | | -----------> |do_page_fault |handle_mm_fault |pte_alloc |pte_alloc_one |__get_free_page = __get_free_pages |alloc_pages |alloc_pages_pgdat |__alloc_pages |wakeup_kswapd // We wake up kernel thread kswapd Page Fault ICA

do_page_fault [arch/i386/mm/fault.c] handle_mm_fault [mm/memory.c] pte_alloc pte_alloc_one [include/asm/pgalloc.h] __get_free_page [include/linux/mm.h] __get_free_pages [mm/page_alloc.c] alloc_pages [mm/numa.c] alloc_pages_pgdat __alloc_pages wakeup_kswapd [mm/vmscan.c]

FIGURES