拨开由问题《linux下malloc最大可申请的内存》带来的重重疑云(Get rid of the many doubts brought by the problem “maximum available memory of malloc under Linux”)

今天阅读相关书籍的时候看到 “进程中堆的最大申请数量” 这一问题,我们知道使用malloc分配内存是在堆Heap里面分配的,如果一台机器一共有8GB物理内存,空闲5GB,那么我们使用malloc( )就一定能够申请到这5GB内存吗?理论上来说确实如此,因为这些内存未被其它进程使用。但实际测试出来结果却可能令人疑惑。

今天阅读相关书籍的时候看到 “进程中堆的最大申请数量” 这一问题,我们知道使用malloc分配内存是在堆Heap里面分配的,如果一台机器一共有8GB物理内存,空闲5GB,那么我们使用malloc( )就一定能够申请到这5GB内存吗?理论上来说确实如此,因为这些内存未被其它进程使用。但实际测试出来结果却可能令人疑惑。

本文测试环境如下:

1 qi@qi:~$ uname -a
2 Linux qi 5.4.0-89-generic #100~18.04.1-Ubuntu SMP Wed Sep 29 10:59:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 

 一、首先需要考虑的几个问题

  • 我们使用malloc( )申请到的是物理内存吗?
  • 使用malloc( )能申请到的只有8GB的物理内存吗?
  • malloc( )申请到的内存大小全都可以被用来memset( )吗?

以上三个问题,正是本次所要讨论的内容。现在假定认为以上三个陈述均正确,那么我们可以用以下程序测试malloc( )可以申请的内存大小:

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 4 
 5 unsigned long long int maximum = 0;
 6 
 7 int main (int argc, char* argv[])
 8 {
 9     unsigned int block_size[] = {1024*1024, 1024, 1};
10     int i, count;
11 
12     for( int i=0; i<3; i++) {
13         for(count=1;; count++) {
14             void *block = malloc(maximum + block_size[i]*count);
15             if( block ) {
16                 //memset(block, 0, maximum + block_size[i]*count);
17                 free(block);
18                 maximum = maximum + block_size[i]*count;
19             } else {
20                 break;
21             }
22         }
23     }
24 
25     printf("maximum malloc size is %llu bytes \n", maximum);
26 
27     return 0;
28 }

 运行以上程序,得到输出为:

1 root@qi:/home/qi/test_park/elf_load# ./main 
2 maximum malloc size is 24587279333 bytes

 可以看到,以上测试程序最大申请到了 22.9GB 的内存,但是我的机器上实际内存有多少呢?如下:

1 qi@qi:~/test_park/elf_load$ free
2               total        used        free      shared  buff/cache   available
3 Mem:        8011016     2373760     3517884      719508     2119372     4654640
4 Swap:      15999996           0    15999996

很明显,机器上最大的物理内存也没到8GB,如果你了解swap 交换空间,可能会说Mem项和Swap项的total加起来似乎正好是22.9GB,但是另外一个问题有来了,那就是这些内存或者交换空间并不是全部空闲,包括系统内核和系统界面等等也要占用一部分物理内存,所以我们看到Mem项的 “available”的可用内存只有大约4.5GB,所以结果就是,malloc( )申请到的内存数量是远远大于我们实际的物理内存的。既然malloc( )函数的实际输出和我们的预期不相符,那是不是我们哪里用错了呢?不妨使用”man malloc”查看对其的官方解释:

1 NOTES
2        By default, Linux follows an optimistic memory allocation strategy.  This means that when malloc() returns non-NULL there is no guarantee that the mem‐
3        ory really is available.  In case it turns out that the system is out of memory, one or more processes will be killed by  the  OOM  killer.   For  more
4        information,  see  the  description  of /proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and the Linux kernel source file Documenta‐
5        tion/vm/overcommit-accounting.

 果不其然,Note中说明了就算malloc( )返回非NULL指针也不能保证该指针指向的内存区域全都可以被该进程使用。那么为什么会这样呢?后面有提示,首先涉及到的最重要的一个设置就是 “/proc/sys/vm/overcommit_memory” 这一个文件,使用 “man proc” 找到有关其的说明:

 1        /proc/sys/vm/overcommit_memory
 2               This file contains the kernel virtual memory accounting mode.  Values are:
 3 
 4                      0: heuristic overcommit (this is the default)
 5                      1: always overcommit, never check
 6                      2: always check, never overcommit
 7 
 8               In mode 0, calls of mmap(2) with MAP_NORESERVE are not  checked,  and  the  default
 9               check is very weak, leading to the risk of getting a process "OOM-killed".
10 
11               In mode 1, the kernel pretends there is always enough memory, until memory actually
12               runs out.  One use case for this mode is  scientific  computing  applications  that
13               employ  large  sparse  arrays.   In Linux kernel versions before 2.6.0, any nonzero
14               value implies mode 1.
15 
16               In mode 2 (available since Linux 2.6), the total virtual address space that can  be
17               allocated (CommitLimit in /proc/meminfo) is calculated as
18 
19                   CommitLimit = (total_RAM - total_huge_TLB) *
20                                 overcommit_ratio / 100 + total_swap

可以看到,如果该文件内容为0,mmap(malloc的内部调用)将不检查,有导致使用不存在内存的风险,如果文件内容为1,则malloc( )可以申请的内存可以非常大,我的机器上经过测试可以达到90T,如果该文件内容为2,那么所有可以申请的内存为 “CommitLimit”,具体可以通过公式或者 “cat /proc/meminfo | grep Limit”查看大小。那么这就能说通为什么上面的程序可以malloc( )出22GB多的内存了,查看 “/proc/sys/vm/overcommit_memory” 果不其然,内容为0:

1 root@qi:/home/qi/test_park/elf_load# cat /proc/sys/vm/overcommit_memory 
2 0

以上回答了第2个问题中的一部分,那就是某些设置下,malloc( )可以申请到超出机器物理内存的大小,为什么说是一部分呢,因为可申请的内存不仅和上述设定相关,还和机器的swap space相关,如果你不了解或者没听过( 事实上在你给你机器装Linux系统的时候应该碰到过,那就是磁盘分区的时候会有一个swap设定)swap空间,只需要知道它是一种挂载在物理硬盘上,用来存放一些不太频繁使用的内存,是一种低速的物理内存的扩展,当物理内存不够用时,原先一些物理内存中不常访问的内容会被转移到这里以让出空间给其它进程。所以swap空间也可以被malloc( )申请到。

由此,第2个问题得到了全部的解答。这个时候你可能会说,第1个问题应该也有答案了,因为malloc( )不仅申请了8GB的物理内存,还申请了15GB的swap硬盘空间作为扩展内存,甚至还可以申请大约90TB的不存在的内存,所以第一个问题就解决了吗?

其实对,但也不全对,因为malloc( )这个时候申请了内存,但没有完全申请,这就涉及到一个叫做 “Lazy Allocation” 的东东,类似于fork的写时复制机制,当你使用malloc( )时,系统并没有真正从物理内存中分配,而是等到进程要操作时才提供allocation,这也就解释了我们刚开头申请了22.9GB的内存都还没有报段错误的原因。只有当你access这个内存区域的时候才会真正分配,所以我们可以大胆的在程序里面加上memset,把上面贴出的代码的memset那一行取消掉注释,然后再运行。如果你不想等太久,可以像我这样:

1 root@qi:/home/qi/test_park/elf_load# echo 0 >  /proc/sys/vm/overcommit_memory 
2 root@qi:/home/qi/test_park/elf_load# swapoff -a
3 root@qi:/home/qi/test_park/elf_load# echo 2 >  /proc/sys/vm/overcommit_memory

以上命令是把交换空间禁用,这样就可以减少可使用的内存了,关闭交换空间后,如果/proc/sys/vm/overcommit_memory内容为0,那么你可以malloc( )的内存大小应该为8GB左右,但是不是每一个字节都可以memset,大可以测试一下,会发现memset了6~7GB的内存空间后程序报错异常退出,这是因为这个时候可使用的内存也就这么大,这种情况下随意使用malloc( )申请到的内存是不安全的。如果/proc/sys/vm/overcommit_memory内容为2,那么这个时候可申请的内存就得看 “CommitLimit” 了,在我的机器上测试是只能申请1.5GB左右,这种情况下无论如何也不会访问非法内存区域了,但是一个缺点是不能使用全部的空闲内存,只能修改相应的设置。

那么该如何知道实际可用的内存大小呢?一种解决方案是查看 “/proc/meminfo” 中的available memory,乘个安全系数再来申请。

以上,三个问题全都被解决,离专业的linuxer又近了一步~

————————

< strong > when reading relevant books today, we saw the problem of “maximum number of requests in the process heap”. We know that malloc is used to allocate memory in heap heap. If a machine has 8GB of physical memory and 5GB of free memory, can we certainly apply for 5GB of memory by using malloc()? In theory, this is true because this memory is not used by other processes. But the actual test results may be puzzling

< strong > when reading relevant books today, we saw the problem of “maximum number of requests in the process heap”. We know that malloc is used to allocate memory in heap heap. If a machine has 8GB of physical memory and 5GB of free memory, can we certainly apply for 5GB of memory by using malloc()? In theory, this is true because this memory is not used by other processes. But the actual test results may be puzzling

< strong > the test environment of this paper is as follows: < / strong >

1 qi@qi:~$ uname -a
2 Linux qi 5.4.0-89-generic #100~18.04.1-Ubuntu SMP Wed Sep 29 10:59:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 

1、 Several problems to be considered first

  • Do we use malloc () to request physical memory?
  • Can only 8GB of physical memory be requested using malloc()?
  • Can all the memory size requested by malloc () be used for memset()?

< strong > the above three questions are exactly the contents to be discussed this time. Now, assuming that the above three statements are correct, we can use the following program to test the memory size that malloc () can apply for: < / strong >

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 4 
 5 unsigned long long int maximum = 0;
 6 
 7 int main (int argc, char* argv[])
 8 {
 9     unsigned int block_size[] = {1024*1024, 1024, 1};
10     int i, count;
11 
12     for( int i=0; i<3; i++) {
13         for(count=1;; count++) {
14             void *block = malloc(maximum + block_size[i]*count);
15             if( block ) {
16                 //memset(block, 0, maximum + block_size[i]*count);
17                 free(block);
18                 maximum = maximum + block_size[i]*count;
19             } else {
20                 break;
21             }
22         }
23     }
24 
25     printf("maximum malloc size is %llu bytes \n", maximum);
26 
27     return 0;
28 }

  Run the above program and the output is: < / strong >

1 root@qi:/home/qi/test_park/elf_load# ./main 
2 maximum malloc size is 24587279333 bytes

  You can see that the maximum memory applied for the above test program is 22.9gb, but how much is the actual memory on my machine? As follows: < / strong >

1 qi@qi:~/test_park/elf_load$ free
2               total        used        free      shared  buff/cache   available
3 Mem:        8011016     2373760     3517884      719508     2119372     4654640
4 Swap:      15999996           0    15999996

< strong > obviously, the largest physical memory on the machine is less than 8GB. If you know the swap swap space, you may say that the total of MEM and swap items seems to be exactly 22.9gb, but another problem is that these memory or swap space are not completely free, including the system kernel and system interface, Therefore, we can see that the “available” memory of the MEM item is only about 4.5gb, so the result is that the amount of memory applied by malloc () is much larger than our actual physical memory. Since the actual output of the malloc () function does not match our expectations, is there something wrong with it? You might as well use “man malloc” to see the official explanation for it: < / strong >

1 NOTES
2        By default, Linux follows an optimistic memory allocation strategy.  This means that when malloc() returns non-NULL there is no guarantee that the mem‐
3        ory really is available.  In case it turns out that the system is out of memory, one or more processes will be killed by  the  OOM  killer.   For  more
4        information,  see  the  description  of /proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and the Linux kernel source file Documenta‐
5        tion/vm/overcommit-accounting.

< Strong > sure enough, the note explains that even if malloc() returns a non null pointer, it cannot guarantee that all the memory areas pointed to by the pointer can be used by the process. So why? You will be prompted later. The most important setting involved first is the file “/ proc / sys / VM / overcommit_memory”. Use “man proc” to find its description: < / strong >

 1        /proc/sys/vm/overcommit_memory
 2               This file contains the kernel virtual memory accounting mode.  Values are:
 3 
 4                      0: heuristic overcommit (this is the default)
 5                      1: always overcommit, never check
 6                      2: always check, never overcommit
 7 
 8               In mode 0, calls of mmap(2) with MAP_NORESERVE are not  checked,  and  the  default
 9               check is very weak, leading to the risk of getting a process "OOM-killed".
10 
11               In mode 1, the kernel pretends there is always enough memory, until memory actually
12               runs out.  One use case for this mode is  scientific  computing  applications  that
13               employ  large  sparse  arrays.   In Linux kernel versions before 2.6.0, any nonzero
14               value implies mode 1.
15 
16               In mode 2 (available since Linux 2.6), the total virtual address space that can  be
17               allocated (CommitLimit in /proc/meminfo) is calculated as
18 
19                   CommitLimit = (total_RAM - total_huge_TLB) *
20                                 overcommit_ratio / 100 + total_swap

< strong > it can be seen that if the file content is 0, MMAP (internal call of malloc) will not be checked, resulting in the risk of using no memory. If the file content is 1, the memory that malloc () can apply for can be very large. After testing on my machine, it can reach 90t. If the file content is 2, all the memory that can be applied for is “commitlimit”, You can check the size through the formula or “cat / proc / meminfo | grep limit”. Then it can explain why the above program can malloc () more than 22gb of memory. Check “< strong > / proc / sys / VM / overcommit_memory”, which is 0: < / strong > < / strong >

1 root@qi:/home/qi/test_park/elf_load# cat /proc/sys/vm/overcommit_memory 
2 0

< strong > the above answers part of the second question, that is, under some settings, malloc () can apply for more than the physical memory of the machine. Why is it part of it? Because the available memory is related not only to the above settings, but also to the swap space of the machine. If you don’t know or haven’t heard of it (in fact, you should have encountered it when installing Linux system on your machine, that is, there will be a swap setting when partitioning the disk) You only need to know that swap space is a kind of memory attached to the physical hard disk to store some less frequently used memory. It is a low-speed expansion of physical memory. When the physical memory is not enough, some infrequently accessed contents in the original physical memory will be transferred here to make room for other processes. Therefore, swap space can also be applied for by malloc(). < / strong >

< strong > thus, the second question has been fully answered. At this time, you may say that the first question should also have an answer, because malloc () not only applies for 8GB of physical memory, but also applies for 15GB of swap hard disk space as extended memory, and even applies for about 90tb of nonexistent memory, so is the first question solved

< strong > in fact, it is true, but not all true, because malloc () applies for memory at this time, but does not fully apply. This involves something called “lazy allocation”, which is similar to fork’s write time replication mechanism. When you use malloc (), the system does not really allocate from physical memory, but provides allocation when the process needs to operate, This explains why we applied for 22.9gb of memory at the beginning, but we haven’t reported segment errors. Only when you access this memory area will it be allocated, so we can boldly add memset to the program, uncomment the memset line of the code posted above, and then run it again. If you don’t want to wait too long, you can do it like me: < / strong >

1 root@qi:/home/qi/test_park/elf_load# echo 0 >  /proc/sys/vm/overcommit_memory 
2 root@qi:/home/qi/test_park/elf_load# swapoff -a
3 root@qi:/home/qi/test_park/elf_load# echo 2 >  /proc/sys/vm/overcommit_memory

< strong > the above command disables the swap space, which can reduce the available memory. After closing the swap space, if / proc / sys / VM / overcommit_ If the memory content is 0, you can tell that the memory size of malloc () should be about 8GB, but not every byte can be memset. You can test it. You will find that after memset has 6 ~ 7GB of memory space, the program will report an error and exit abnormally. This is because the available memory is so large at this time. In this case, it is unsafe to use the memory applied by malloc () at will< Strong > if / proc / sys / VM / overcommit_ If the memory content is 2, the memory that can be applied for at this time depends on “< strong > commitlimit < / strong >“. The test on my machine can only apply for about 1.5GB. In this case, the illegal memory area will not be accessed anyway, but one disadvantage is that you can’t use all the free memory and can only modify the corresponding settings

< strong > < strong > How do you know the actual available memory size? One solution is to check the available memory in “/ proc / meminfo” and multiply the safety factor to apply again

< strong > < strong > all the above three problems have been solved, which is a step closer to professional Linuxer ~ < / strong > < / strong >