Memory Reservation Pools in Oracle Solaris 11.4 SRU1

There is a rare sitation that prohibits you to boot or reboot a Kernel Zone. In order to happen, you need to have a massively loaded system and a lot is going on on your system. The problem is, that you don’t have enough memory in a way the Solaris Kernel Zones can use. Furthermore: You may remember that the documentation suggests to set the user_reserve_hint_pct parameter in order to ensure that a ZFS ARC can’t grow that large, that you can’t start the kernel zone.

In order to improve on both situations and make the first phenomenon even more rare than it is currently, a new feature was introduced into Solaris 11.4. It’s called memory reservation pools (MRP). However you can’t use them in the generally available image, as it was introduced with SRU1 and is thus an example of our policy to deliver new features with SRU as well. So, you need access to the SRU in order to try and use this feature.

The basic idea is simple: You reserve memory very early while booting and afterwards you can tell a kernel zone not to allocate its memory from the general memory, but from this reserved pool of memory.

How do you configure it?

It’s quite simple to use. However let us check first the current memory situation.

root@batou:~# echo "::memstat" | mdb -k
Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                                 253718   991.0m 12.0%
ZFS                                    174134   680.2m  8.3%
User/Anon                              100588   392.9m  4.7%
Exec and libs                            2172     8.4m  0.1%
Page Cache                              11054    43.1m  0.5%
Free (cachelist)                        16991    66.3m  0.8%
Free                                  1538382     5.8g 73.3%
Total                                 2097039     7.9g  100%

MRP are controlled by an SMF service. Thus we configure it with some svccfg statements.

root@batou:~# svccfg -s svc:/system/memory-reserve:zones setprop config/size=1g
root@batou:~#svccfg -s svc:/system/memory-reserve:zones setprop config/type=solaris-kz
root@batou:~#svccfg -s svc:/system/memory-reserve:zones setprop config/lgrps=""
root@batou:~#svccfg -s svc:/system/memory-reserve:zones refresh 
root@batou:~#svcadm enable svc:/system/memory-reserve:zones

When you check for the memory situation again, you will see a new line in the output:

root@batou:~# echo "::memstat" | mdb -k
Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                                 255036   996.2m 12.1%
ZFS                                    177220   692.2m  8.4%
User/Anon                              101019   394.6m  4.8%
Exec and libs                            2176     8.5m  0.1%
Page Cache                              11016    43.0m  0.5%
Free (cachelist)                        18386    71.8m  0.8%
Free                                  1532186     5.8g 73.0%
  Regular Free                        1270042     4.8g       60.5%/82.8%
  MRP reserved mem                     262144       1g       12.5%/17.1%
Total                                 2097039     7.9g  100%

The size of the reserved pool can be changed without an reboot

root@batou:~# echo "::memstat" | mdb -k | grep "MRP"
  MRP reserved mem                     262144       1g       12.5%/17.1%
root@batou:~#  svccfg -s svc:/system/memory-reserve:zones setprop config/size=2g
root@batou:~#  svccfg -s svc:/system/memory-reserve:zones refresh
root@batou:~# echo "::memstat" | mdb -k | grep "MRP"
  MRP reserved mem                     524288       2g       25.0%/34.2%

Okay, you have reserved some memory, but how do you use it? The only consumer for such reserved memory are kernel zones, thus you configure its use in the zone configuration.

root@batou:~# zonecfg -z kzone1 info capped-memory
 capped-memory:
        physical: 2G
        pagesize-policy: largest-available
root@batou:~# zonecfg -z kzone1 'select capped-memory; clear pagesize-policy; set memory-reserve=zones; end; commit'

That's pretty much all. In the memory-reserve property you have to enter the name of the instance of the svc:/system/memory-reserve service that created the MRP , thus when using the default svc:/system/memory-reserve:zones the MRP name you have to use is zones

The best practive is from now on, that you don't use user_reserve_hint_pct to ensure that enough memory is free for Kernel Zones, but by using memory reservation pools.

Do you want to learn more?

support.oracle.com - Oracle Solaris 11.4.1.4.0 Support Repository Update (Oracle Solaris 11.4 SRU 1.4.0) (Doc ID 2449090.1)