Overview

The purpose of this lab assignment is to conduct experiments that introduce you to virtual memory concepts. Specifically, we will experiment with the effect of memory-intensive process execution, frame allocation policies, and page replacement algorithms by observing periodic system reports of virtual memory statistics.

Activities

• Work your way through the following exercises, demonstrating your knowledge of the material by answering the numbered questions.

• Submit a detailed lab report that includes your answers to the numbered questions. Think about what these experiments are trying to discover/illustrate.

Virtual Memory Performance Statistics

Self-monitoring is a task performed by most operating systems — it is useful for administration and auditing purposes. Since there are many subsystems in a typical OS, there are many different performance auditing programs. This lab concerns virtual memory, hence the primary utility tool used will be vmstat (virtual memory statistics). The utility programs free and top also give memory usage statistics. Begin by reading the man pages for these utilities to get an idea of the types of information they provide and the various output options available via command-line parameters.

The vmstat utility is typically used by system administrators to characterize the performance of a system. For example, to an administrator a large number of swapped processes and a large amount of active virtual memory is indicative of a system with a severe memory shortage (and high paging activity), hence it's time to install more RAM.

TIP

There are several different types of memory, and memory measures, referred to in this lab. Be sure you understand the difference and the correct use of each metric. Also, pay special attention to the units; i.e. does a utility report memory usage in bytes, kilobytes (KB), megabytes (MB), etc.

Determine your system configuration

1. Specify which EOS or Arch workstation you are working on.

IMPORTANT

Be sure to use the same EOS workstation throughout the experiments below. Our arch/EOS machines do not have the same memory configurations.

2. Use the following utilities to obtain memory-related information of your system:

   free --giga
   # Also try free --mega  and free --kilo
   getconf -a | grep -i page

   What is:

   • the total amount of physical memory on your system (in Gbytes)?
   • the total amount of swap space on your system (in Gbytes)?
   • the current amount of free memory (in GBytes)?
   • the page size used by your system (in bytes)?

Sample Program 1

Study the following sample program, then compile and run it.

[Gist]

API rate limit exceeded for 23.23.19.54. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

TIP

If the program ran without failing memory allocation, change the for-loop upper limit until it failed the memory allocation.

3. What is the biggest amount of heap successfully allocated by the program? Was the program able to allocate more than the (free) physical RAM? Relate your answer to the values obtained in question (2).

Examine and Observe Memory Demand

Study the following test program; in particular, noting its memory requirements and its memory access profile. Note, the value dim, expressed in Kilobytes, is used to control memory demand by establishing a dim X dim matrix of integers. The value LOOP is used to keep the program executing long enough to observe its memory demand.

Sample Program 2

[Gist]

API rate limit exceeded for 23.23.19.54. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

Study Sample Program 2, then compile it and run it once (to prime the system).

4. What is your estimate of the memory demand of the Sample Program?

   • in kilo bytes or mega bytes _______
   • in number of pages _________

Read the man page of vmstat to see some details about using the command and its multi-column output.

• Start vmstat and configure it to display statistics once per second.
• Let the system stabilize.
• Uncomment the usleep() call, and recompile the program. On faster CPUs the sample program runs much faster than the vmstat can track memory use per second. The extra usleep() call attempts to slow down your program allowing vmstat to capture the memory use. Choose a spot for placing the usleep() call (outermost, middle, or innermost for-loop)
• In another window, execute the Sample Program 2 again.

5. Approximately how much does the amount of free (idle) memory change while the above program is running?

   • considering your computed memory demands of Sample Program 2, explain why the observed change is a predictable result.

   TIP

   While your program is running, the output of vmstat may keep changing. Wait until the output stabilizes to determine the amount of memory change. One of command line options accepted by vmstat is --unit. Use this option so it is easier for you to observe the amount of memory in either Kilobyte or Megabyte.

Examine the effect of Increased Demand

Increase the memory allocation by replacing the initial value of AMOUNT to a bigger number that causes the total memory demand of Sample Program 2 to be just less than the amount memory (excluding swap) on your machine. The objective is to deplete the free memory as close as possible to zero WITHOUT failing the program.

Note: the computed value for AMOUNT will differ on different machines.

Changing AMOUNT will only allow you to make changes in multiple of 1024 bytes. If you need to make finer adjustment to the allocation amount, you can also change the value of dim from AMOUNT * KILO_BYTE to a value which is not a multiple of 1024.

6. Describe and justify your choice of parameter value.

   TIP

   • your total demand should be a little less than the observed amount of idle memory (as indicated by vmstat)
   • your total memory demand should not exceed the total amount of physical memory on your system (as determined in question 2).
   • reduce the LOOP parameter to 1 (or prepare to wait a while).

Configure and run vmstat to display statistics once per second. In another window, execute Sample Program 2.

7. Observe what happens to the amount of idle memory. Given your choice of dim from the previous step, is this what you expected to see? Why / why not?

8. Reference the man page for vmstat to understand what is exactly being displayed. What other memory field(s) change (i.e. swap device, buffers, cache)? Speculate: why might these fields have changed (i.e. explain how the system adapting to the increased memory demand)

Examine the effect of memory access patterns

Change the dim back to their original values and set LOOP to 1. Read the man pages for the time utility program. Then use /usr/bin/time (with command-line arguments as described for time) to obtain complete statistics (i.e. verbose mode) for execution of Sample Program 2. Run and time Sample Program 2 and obtain statistics.

9. What is the difference between a major and a minor page fault in Linux (you might need to look this up)?

Comment out the usleep() call and change the memory access statements in Sample Program 2 to read:

void* addr = intPtr + j*dim + i;
intPtr[j * dim + i] = (i + j) % count;

and re-compile the program.

10. Precisely, how does this change alter the program's memory access pattern (i.e. what objects get "touched", in what order)?

HINTS

• uncomment the printf() statement that shows the visited memory address and its page number. Compare the print() output before and after the above changes. Recall your understanding of C pointer arithmetic to understand the output of this printf()
• use the total size of the (2D) array and the Linux page size to draw a sketch of the memory layout of the array elements and how they are stored across multiple pages in your logical address space.
• Comment out the printf() statement before you continue with the next question below

1. Speculate: how will this change affect the program's execution time (verify your answer by executing and timing the program, be sure to comment out the printf() statement in the innermost loop)?

2. Precisely, why does the change have this effect (your answer should incorporate an important concept discussed in class)?

Examine the use of virtual memory

Change the memory access pattern for Sample Program 2 back to its original form. Change the LOOP value to 1. Adjust the dim parameter in Sample Program 2 to a value that causes the total memory demand of the program to exceed the total amount of physical memory on your machine (as determined in question 2 above), but without failing memory allocation.

13. Describe and justify your computation.

Configure and run vmstat to display statistics once every second. Run sample program 2 again.

14. Describe what has happened and why (i.e. explain how the operating system is adapting to the increased memory demands of Sample Program 2).

IMPORTANT

Include a brief discussion of the execution time and number of page faults incurred. Your explanation should demonstrate that you understand what is happening from a virtual memory viewpoint.