Thursday, July 21, 2011

Linux Memory Management

[root@vmlx-106 /var/www/html]# ps -A --sort -rss -o comm,pmem | head -n 11
COMMAND         %MEM
gdmgreeter       1.6
httpd            1.3
httpd            0.8
httpd            0.7
httpd            0.6
httpd            0.6
httpd            0.6
httpd            0.6
httpd            0.6
httpd            0.6
[root@vmlx-106 /var/www/html]# while read command percent rss; do if [[ "${command}" != "COMMAND" ]]; then rss="$(bc <<< "scale=2;${rss}/1024")"; fi; printf "%-26s%-8s%s\n" "${command}" "${percent}" "${rss}"; done < <(ps -A --sort -rss -o comm,pmem,rss | head -n 11)
COMMAND                   %MEM    RSS
gdmgreeter                1.6     16.89
httpd                     1.3     13.75
httpd                     0.8     8.26
httpd                     0.7     7.49
httpd                     0.6     6.94
httpd                     0.6     6.94
httpd                     0.6     6.94
httpd                     0.6     6.94
httpd                     0.6     6.94
httpd                     0.6     6.94
[root@vmlx-106 /var/www/html]# free -m
             total       used       free     shared    buffers     cached
Mem:          1002        723        278          0        174        408
-/+ buffers/cache:        140        861
Swap:         1027          0       1027

I see your using the command I mentioned above but with one less item in the head command output but that’s alright ;)

You can show the same data, without limiting the input but also displaying the RSS usage (and the pid) for each process by running:

ps -A –sort -rss -o pid,comm,pmem,rss


The far right column will show you the RSS usage in kilobytes. The order is descending so the application with the highest allocated RSS (using the most) is at the top of the list. The application with the lowest allocated RSS (using the least) is at the bottom of the list. Remember that RSS is only referring to the memory you application is using that is actually in RAM and does not include anything swapped to disk so this is in fact the amount of RAM your application is using.

Something important though that you may not have realized is that Linux tends to use almost all available memory whenever it can by default. Linux will leave an percent free (I don’t know how to check what the percent is but it doesn’t matter) and then it will allocate the rest to buffers and cache. The general idea is this, buffers is the smaller segment it allocates for I/O usage of processes that are running. This mostly refers to items that programs need to write to a disk but that data is often cached in memory and then written. This causes programs not to have to wait for I/O to complete and the kernel is then able to write the data to disk as quickly as it’s capable of doing but at the same time without having to have a program and hence the user wait for the information to be written to disk. The buffers are also used for IPC or inter process communication. The cache is mostly used for reading files off disk. The algorithm is uses, while complex, basically stores a combination of files that are popular and recent in cache memory. This speeds up disk read times a lot too. You will notice that the cache is almost always significantly larger then the buffers. You can view both of these using free to see total, used and free memory. You can use free -m for megabytes and free -g for gigabytes.


This RAM, more quickly with cache then buffers, can be very quickly (on both) be released and therefor given to any process asking for RAM. This means that, although this RAM is used and it is used in way that speeds up your system, if any task needs the RAM, it will instantly be given to that process. One example would be if I start VLC movie player and VLC needs and therefor requests 50MB to run, then the kernel will free 50MB from cache and buffers and allocate it to VLC. This is why there is also a line when you run free which says “-/+ buffers/cache”. This line which shows numbers under used and free, isn’t the RAM you literally have used and literally have free because of ram allocated to cache and buffers, however it is the line you should refer to when you want to see how much free RAM you have because RAM allocated to buffers and cache will be instantly released and given to a process whenever it needs it.


When you view your free memory inside top, it shows you the literally used RAM and literally free RAM meaning cache and buffers are added to used and subtracted from free and their is nothing wrong with this but the RAM inside cache and buffers is still available to you when you need it. As I mentioned, the best way to really see how much used and free RAM you have is to look at the “-/+ buffers/cache” line inside free. That line is there for a reason.

rsync with delete option and different ssh port

How to rsync e.g PIPELINE dir from Source to Destination? #rsync -avzr   --delete-before  -e "ssh -p $portNumber"  /local...