案例：ORA-04031 12.1.0.2 on exadata x7

数据和云

发布于 2022-02-25 16:54:45

4920

某银行客户近期频繁出现ORA-04031，报错如下:

[TOC00000]Jump to table of contentsDump continued from file: /u01/app/oracle/diag/rdbms/coredb/coredb1/trace/coredb1_ora_291223.trc[TOC00001]ORA-04031: unable to allocate 920 bytes of shared memory ("shared pool","SELECT                 ACC...","SQLA^559c65a1","qeeRwo: qeeCreateRwo")
[TOC00001-END]

此次报错的subheap为SQLA^559c65a1，通过trace查看该子堆与父堆的信息。

******************************************************HEAP DUMP heap name="sga heap"  desc=0x600013d0 extent sz=0x4d3a8 alt=272 het=32767 rec=9 flg=130 opc=0 parent=(nil) owner=(nil) nex=(nil) xsz=0x1 heap=(nil) fl2=0x60, nex=(nil)  pdb id=0 ds for latch 1: 0x600884c8                                  ds for latch 2: 0x60092218                                  ds for latch 3: 0x6009bf68                                  ds for latch 4: 0x600a5cb8                                  ds for latch 5: 0x600afa08                                  ds for latch 6: 0x600b9758                                  ds for latch 7: 0x600c34a8                                  reserved granule count 0 (granule size 33554432)******************************************************HEAP DUMP heap name="SQLA^559c65a1"  desc=0x192d7c9b00 extent sz=0xfe8 alt=32767 het=456 rec=0 flg=2 opc=2 parent=0x600013d0 owner=0x192d7c99b8 nex=(nil) xsz=0x2000000 heap=(nil) fl2=0x27, nex=(nil), dsxvers=1, dsxflg=0x0 dsx first ext=0x1a0df95090 dsx empty ext bytes=0  subheap rc link=0x1a0df95100,0x1a0df95100 pdb id=0 Subheap has 85360 bytes of memory allocated

可以看到该子堆已经分配了85360 bytes，extent大小为4072。

根据chunk的分配原则，首先尝试在子堆中分配，如果子堆free list无法分配，则向父堆申请。这时申请的chunk大小将会是extent size，并作为一个新的extent挂在子堆下。

查看子堆的free list如下：

/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释 
* HEAP DUMP heap name="SQLA^559c65a1"  desc=0x192d7c9b00 extent sz=0xfe8 alt=32767 het=456 rec=0 flg=2 opc=2 parent=0x600013d0 owner=0x192d7c99b8 nex=(nil) xsz=0x2000000 heap=(nil) fl2=0x27, nex=(nil), dsxvers=1, dsxflg=0x0 dsx first ext=0x1a0df95090 dsx empty ext bytes=0  subheap rc link=0x1a0df95100,0x1a0df95100 pdb id=0EXTENT 0 addr=0x1933b5dfc8  Chunk       1933b5dfd8 sz=      704    free      "               "Dump of memory from 0x0000001933B5DFD8 to 0x0000001933B5E2981933B5DFD0                   000002C1 C0B38F00          [........]1933B5DFE0 00000000 00000000 2D7C9BD8 00000019  [..........|-....]1933B5DFF0 2D7C9BD8 00000019 01010101 00000000  [..|-............]1933B5E000 00000000 00000000 0EB447B8 00000000  [.........G......]1933B5E010 00000000 00000070 00000001 C0B38F00  [....p...........]1933B5E020 00000000 00000000 33B5E948 00000019  [........H..3....]1933B5E030 2F36F320 00000019 00000001 00000000  [ .6/............]1933B5E040 33B5DFE0 00000019 00000000 00000000  [...3............]1933B5E050 33B5E050 00000019 33B5E050 00000019  [P..3....P..3....]1933B5E060 000008D9 40B38F00 33B5DFF0 00000019  [.......@...3....]1933B5E070 33B5DFF0 00000019 0EBBECFC 00000000  [...3............]1933B5E080 000002A0 00000328 9DF614B8 00000019  [....(...........]1933B5E090 33B5E200 00000019 00000000 00000000  [...3............]1933B5E0A0 2F36F258 00000019 00000000 00000000  [X.6/............]1933B5E0B0 00000000 00000000 00000000 00000000  [................]1933B5E0C0 33B5E150 00000019 00000000 00000000  [P..3............]1933B5E0D0 00000000 00000000 00000000 00000000  [................]        Repeat 7 times1933B5E150 33B5E300 00000019 00000000 00000000  [...3............]1933B5E160 00000000 00040340 00000000 00000000  [....@...........]1933B5E170 00000000 00000000 33B5E088 00000019  [...........3....]1933B5E180 00000000 00000000 00000004 00000000  [................]1933B5E190 00000000 00000000 D4881520 0000001A  [........ .......]1933B5E1A0 00000006 00000010 00000000 00000000  [................]1933B5E1B0 00000000 00000000 00000000 00000000  [................]1933B5E1C0 000018CA FFFFFFFF 00000000 00000000  [................]1933B5E1D0 00000000 00000000 00000000 00000000  [................]        Repeat 1 times1933B5E1F0 8938A048 161BEF84 ECDBBD88 2D39B55E  [H.8.........^.9-]1933B5E200 33B5E2A0 00000019 00000010 00000000  [...3............]1933B5E210 00000004 00000000 00000000 00000000  [................]1933B5E220 00000000 00000000 00000000 00000000  [................]        Repeat 6 times1933B5E290 00000000 00000000                    [........]          Chunk       1933b5e298 sz=      928    freeable  "qeeRwo: qeeCrea"...FREE LISTS: Bucket 0 size=24  Chunk       1a0df950c8 sz=        0    kghdsx Bucket 1 size=88 Bucket 2 size=152 Bucket 3 size=280 Bucket 4 size=536  Chunk       1933b5dfd8 sz=      704    free      "               "Dump of memory from 0x0000001933B5DFD8 to 0x0000001933B5E2981933B5DFD0                   000002C1 C0B38F00          [........]1933B5DFE0 00000000 00000000 2D7C9BD8 00000019  [..........|-....]1933B5DFF0 2D7C9BD8 00000019 01010101 00000000  [..|-............]1933B5E000 00000000 00000000 0EB447B8 00000000  [.........G......]1933B5E010 00000000 00000070 00000001 C0B38F00  [....p...........]1933B5E020 00000000 00000000 33B5E948 00000019  [........H..3....]1933B5E030 2F36F320 00000019 00000001 00000000  [ .6/............]1933B5E040 33B5DFE0 00000019 00000000 00000000  [...3............]1933B5E050 33B5E050 00000019 33B5E050 00000019  [P..3....P..3....]1933B5E060 000008D9 40B38F00 33B5DFF0 00000019  [.......@...3....]1933B5E070 33B5DFF0 00000019 0EBBECFC 00000000  [...3............]1933B5E080 000002A0 00000328 9DF614B8 00000019  [....(...........]1933B5E090 33B5E200 00000019 00000000 00000000  [...3............]1933B5E0A0 2F36F258 00000019 00000000 00000000  [X.6/............]1933B5E0B0 00000000 00000000 00000000 00000000  [................]1933B5E0C0 33B5E150 00000019 00000000 00000000  [P..3............]1933B5E0D0 00000000 00000000 00000000 00000000  [................]        Repeat 7 times1933B5E150 33B5E300 00000019 00000000 00000000  [...3............]1933B5E160 00000000 00040340 00000000 00000000  [....@...........]1933B5E170 00000000 00000000 33B5E088 00000019  [...........3....]1933B5E180 00000000 00000000 00000004 00000000  [................]1933B5E190 00000000 00000000 D4881520 0000001A  [........ .......]1933B5E1A0 00000006 00000010 00000000 00000000  [................]1933B5E1B0 00000000 00000000 00000000 00000000  [................]1933B5E1C0 000018CA FFFFFFFF 00000000 00000000  [................]1933B5E1D0 00000000 00000000 00000000 00000000  [................]        Repeat 1 times1933B5E1F0 8938A048 161BEF84 ECDBBD88 2D39B55E  [H.8.........^.9-]1933B5E200 33B5E2A0 00000019 00000010 00000000  [...3............]1933B5E210 00000004 00000000 00000000 00000000  [................]1933B5E220 00000000 00000000 00000000 00000000  [................]        Repeat 6 times1933B5E290 00000000 00000000                    [........]         Bucket 5 size=1048 Bucket 6 size=2072 Bucket 7 size=4120 Bucket 8 size=4144 Bucket 9 size=4168 Bucket 10 size=4192 Bucket 11 size=4216Total free space   =      704
*/

请求940字节，但是子堆free list只有704字节，所以这时会向父堆申请一个子堆extent size的chunk，即4072字节，但是trace中并未发现父堆的详细信息。

Did Not dump SGA Heap desc=0x600013d0.   Current time: 10/26/2021 04:13:50  Last SGA heap dump at 10/26/2021 04:13:38  SGA Heap Dump Count = 2

查看subpool的top组件：

/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释 
* ==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 1----------------------------------------------"ksfqpn                    "        430 MB 30%"gcs resources             "        375 MB 26%"gcs shadows               "        207 MB 14%"db_block_hash_buckets     "        100 MB  7%"free memory               "         93 MB  6%"init_heap_kfsg            "         48 MB  3%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%"object queue hash buckets "         14 MB  1%"Checkpoint queue          "         13 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 1----------------------------------------------"ksfqpn                    "        430 MB 26%"gcs resources             "        375 MB 23%"gcs shadows               "        207 MB 13%"SQLA                      "        132 MB  8%"free memory               "        114 MB  7%"db_block_hash_buckets     "        100 MB  6%"init_heap_kfsg            "         48 MB  3%"KGLH0                     "         46 MB  3%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%     -----------------------------------------free memory                         207 MBmemory alloc.                      2880 MBSub total                          3086 MB==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 2----------------------------------------------"ksfqpn                    "        430 MB 29%"gcs resources             "        375 MB 25%"gcs shadows               "        207 MB 14%"db_block_hash_buckets     "        102 MB  7%"free memory               "         92 MB  6%"SQLA                      "         36 MB  2%"FileOpenBlock             "         24 MB  2%"KGLH0                     "         20 MB  1%"gcs res hash bucket       "         16 MB  1%"dbktb: trace buffer       "         15 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 2----------------------------------------------"ksfqpn                    "        430 MB 26%"gcs resources             "        375 MB 22%"gcs shadows               "        207 MB 12%"SQLA                      "        168 MB 10%"free memory               "        114 MB  7%"db_block_hash_buckets     "        102 MB  6%"KGLH0                     "         52 MB  3%"FileOpenBlock             "         24 MB  1%"gcs res hash bucket       "         16 MB  1%"dbktb: trace buffer       "         15 MB  1%     -----------------------------------------free memory                         206 MBmemory alloc.                      2944 MBSub total                          3150 MB==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 3----------------------------------------------"ksfqpn                    "        430 MB 29%"gcs resources             "        375 MB 25%"gcs shadows               "        209 MB 14%"free memory               "        109 MB  7%"db_block_hash_buckets     "        101 MB  7%"SQLA                      "         20 MB  1%"gc name table             "         18 MB  1%"gcs res hash bucket       "         16 MB  1%"dbktb: trace buffer       "         15 MB  1%"KGLH0                     "         14 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 3----------------------------------------------"ksfqpn                    "        430 MB 26%"gcs resources             "        375 MB 22%"gcs shadows               "        209 MB 12%"free memory               "        136 MB  8%"SQLA                      "        133 MB  8%"db_block_hash_buckets     "        101 MB  6%"KGLH0                     "         47 MB  3%"KQR L PO                  "         23 MB  1%"gc name table             "         18 MB  1%"gcs res hash bucket       "         16 MB  1%     -----------------------------------------free memory                         245 MBmemory alloc.                      2908 MBSub total                          3153 MB==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 4----------------------------------------------"ksfqpn                    "        430 MB 29%"gcs resources             "        375 MB 25%"gcs shadows               "        210 MB 14%"free memory               "        111 MB  7%"db_block_hash_buckets     "        100 MB  7%"SQLA                      "         44 MB  3%"KGLH0                     "         22 MB  1%"gcs res hash bucket       "         16 MB  1%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 4----------------------------------------------"ksfqpn                    "        430 MB 25%"gcs resources             "        375 MB 22%"gcs shadows               "        210 MB 12%"SQLA                      "        176 MB 10%"free memory               "        131 MB  8%"db_block_hash_buckets     "        100 MB  6%"KGLH0                     "         56 MB  3%"gcs res hash bucket       "         16 MB  1%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%     -----------------------------------------free memory                         242 MBmemory alloc.                      2971 MBSub total                          3213 MB==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 5----------------------------------------------"ksfqpn                    "        430 MB 30%"gcs resources             "        375 MB 26%"gcs shadows               "        210 MB 15%"db_block_hash_buckets     "        101 MB  7%"free memory               "         96 MB  7%"SQLA                      "         17 MB  1%"KGLH0                     "         15 MB  1%"KQR X SO                  "         15 MB  1%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 5----------------------------------------------"ksfqpn                    "        430 MB 26%"gcs resources             "        375 MB 23%"gcs shadows               "        210 MB 13%"SQLA                      "        129 MB  8%"free memory               "        115 MB  7%"db_block_hash_buckets     "        101 MB  6%"KGLH0                     "         46 MB  3%"KQR X SO                  "         38 MB  2%"KQR X PO                  "         18 MB  1%"dbktb: trace buffer       "         15 MB  1%     -----------------------------------------free memory                         211 MBmemory alloc.                      2879 MBSub total                          3090 MB==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 6----------------------------------------------"ksfqpn                    "        430 MB 30%"gcs resources             "        375 MB 26%"gcs shadows               "        208 MB 14%"db_block_hash_buckets     "        100 MB  7%"free memory               "         92 MB  6%"keomg: entry list         "         36 MB  3%"ges big msg buffers       "         30 MB  2%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%"object queue hash buckets "         14 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 6----------------------------------------------"ksfqpn                    "        430 MB 23%"gcs resources             "        375 MB 20%"free memory               "        233 MB 12%"gcs shadows               "        208 MB 11%"PMR sga heap              "        134 MB  7%"SQLA                      "        128 MB  7%"db_block_hash_buckets     "        100 MB  5%"KGLH0                     "         47 MB  2%"keomg: entry list         "         36 MB  2%"ges big msg buffers       "         30 MB  2%     -----------------------------------------free memory                         325 MBmemory alloc.                      3021 MBSub total                          3346 MB==============================================TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 7----------------------------------------------"ksfqpn                    "        430 MB 29%"gcs resources             "        375 MB 25%"gcs shadows               "        207 MB 14%"db_block_hash_buckets     "        101 MB  7%"free memory               "         94 MB  6%"FileOpenBlock             "         24 MB  2%"KQR L PO                  "         18 MB  1%"SQLA                      "         16 MB  1%"dbktb: trace buffer       "         15 MB  1%"ASH buffers               "         14 MB  1%==============================================TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 7----------------------------------------------"ksfqpn                    "        430 MB 23%"gcs resources             "        375 MB 20%"gcs shadows               "        207 MB 11%"free memory               "        204 MB 11%"SQLA                      "        138 MB  7%"PMR sga heap              "        134 MB  7%"db_block_hash_buckets     "        101 MB  5%"KGLH0                     "         47 MB  2%"FileOpenBlock             "         24 MB  1%"KQR L PO                  "         18 MB  1%     -----------------------------------------free memory                         298 MBmemory alloc.                      3069 MBSub total                          3367 MBTOTALS ---------------------------------------Total free memory                  1734 MBTotal memory alloc.                  20 GBGrand total                          22 GB
*/

发现组件ksfqpn的内存占用异常，gcs resources和gcs shadows也占用偏高。

ksfqpn与IO接口相关，ksfq – kernel service functions sequential file io interface，通过查阅MOS文档发现与Bug 25058080 Excessive allocation of memory “ksfqpn” in SGA using RMAN backups in Exadata匹配。

其bug主要原因是内部算法导致ksfqpn内存重用和回收出现了异常，导致ksfqpn大小是x$ksfgp的1000倍以上。

关于作者李翔宇，云和恩墨西区交付技术顾问，长期服务移动运营商行业客户，熟悉Oracle性能优化，故障诊断，特殊恢复。

本文参与腾讯云自媒体同步曝光计划，分享自微信公众号。

原始发表：2022-01-11，如有侵权请联系 cloudcommunity@tencent.com 删除

编程算法

本文分享自数据和云微信公众号，前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体同步曝光计划，欢迎热爱写作的你一起参与！

编程算法

登录后参与评论

0 条评论

热度