问使用tensorflow进行迭代过程出现failed to allocate memory的bug？

提问于 2021-10-21 22:06:36

回答 1关注 0查看 2.4K

使用tensorflow进行全连接网络迭代过程出现failed to allocate memory的报错

具体情况如下：

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:235: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features.

warnings.warn(

D:\IDE\python\310\lib\site-packages\sklearn\preprocessing\_data.py:254: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0.

warnings.warn(

2021-10-21 21:58:27.722550: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2021-10-21 21:58:29.628655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3989 MB memory: -> device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5

warnings.warn(

2021-10-21 21:59:06.543918: W tensorflow/core/common_runtime/bfc_allocator.cc:457] Allocator (GPU_0_bfc) ran out of memory trying to allocate 7.54GiB (rounded to 8100000000)requested by op Sub

If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.

Current allocation summary follows.

2021-10-21 21:59:06.544339: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] BFCAllocator dump for GPU_0_bfc

2021-10-21 21:59:06.544465: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (256): Total Chunks: 26, Chunks in use: 24. 6.5KiB allocated for chunks. 6.0KiB in use in bin. 780B client-requested in use in bin.

2021-10-21 21:59:06.544711: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (512): Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.544986: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.

2021-10-21 21:59:06.545207: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.545419: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.554742: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.555085: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.555385: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.555684: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (65536): Total Chunks: 2, Chunks in use: 2. 193.0KiB allocated for chunks. 193.0KiB in use in bin. 192.9KiB client-requested in use in bin.

2021-10-21 21:59:06.555973: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (131072): Total Chunks: 2, Chunks in use: 2. 352.0KiB allocated for chunks. 352.0KiB in use in bin. 351.6KiB client-requested in use in bin.

2021-10-21 21:59:06.556604: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.556855: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (524288): Total Chunks: 1, Chunks in use: 1. 879.0KiB allocated for chunks. 879.0KiB in use in bin. 878.9KiB client-requested in use in bin.

2021-10-21 21:59:06.557106: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.557337: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.557573: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.558275: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.558524: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.560650: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.560917: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.561147: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.561888: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 3.89GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.

2021-10-21 21:59:06.562149: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Bin for 7.54GiB was 256.00MiB, Chunk State:

2021-10-21 21:59:06.562539: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 4183621632

2021-10-21 21:59:06.563063: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70ba00000 of size 900096 next 1

2021-10-21 21:59:06.563196: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70badbc00 of size 1280 next 2

2021-10-21 21:59:06.563319: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70badc100 of size 97536 next 3

2021-10-21 21:59:06.563477: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70baf3e00 of size 100096 next 4

2021-10-21 21:59:06.563604: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c500 of size 256 next 5

2021-10-21 21:59:06.563726: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c600 of size 256 next 6

2021-10-21 21:59:06.563849: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c700 of size 256 next 7

2021-10-21 21:59:06.564386: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c800 of size 256 next 8

2021-10-21 21:59:06.564510: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0c900 of size 256 next 9

2021-10-21 21:59:06.564632: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0ca00 of size 256 next 10

2021-10-21 21:59:06.564788: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cb00 of size 256 next 11

2021-10-21 21:59:06.564913: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cc00 of size 256 next 12

2021-10-21 21:59:06.565035: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0cd00 of size 256 next 13

2021-10-21 21:59:06.565166: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0ce00 of size 256 next 14

2021-10-21 21:59:06.565859: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0cf00 of size 256 next 15

2021-10-21 21:59:06.565999: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d000 of size 256 next 16

2021-10-21 21:59:06.566148: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d100 of size 256 next 17

2021-10-21 21:59:06.566271: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d200 of size 256 next 18

2021-10-21 21:59:06.566394: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0d300 of size 256 next 19

2021-10-21 21:59:06.566533: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d400 of size 256 next 20

2021-10-21 21:59:06.566665: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d500 of size 256 next 21

2021-10-21 21:59:06.566788: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d600 of size 256 next 22

2021-10-21 21:59:06.566926: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d700 of size 256 next 23

2021-10-21 21:59:06.576100: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d800 of size 256 next 24

2021-10-21 21:59:06.576228: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0d900 of size 256 next 25

2021-10-21 21:59:06.576349: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0da00 of size 256 next 26

2021-10-21 21:59:06.576476: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0db00 of size 256 next 27

2021-10-21 21:59:06.576629: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0dc00 of size 256 next 28

2021-10-21 21:59:06.576754: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb0dd00 of size 512 next 30

2021-10-21 21:59:06.577267: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0df00 of size 256 next 31

2021-10-21 21:59:06.577415: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0e000 of size 256 next 32

2021-10-21 21:59:06.577542: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb0e100 of size 180224 next 29

2021-10-21 21:59:06.577680: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 70bb3a100 of size 180224 next 33

2021-10-21 21:59:06.577831: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 70bb66100 of size 4182155008 next 18446744073709551615

2021-10-21 21:59:06.577984: I tensorflow/core/common_runtime/bfc_allocator.cc:1065] Summary of in-use Chunks by size:

2021-10-21 21:59:06.578122: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 24 Chunks of size 256 totalling 6.0KiB

2021-10-21 21:59:06.578255: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 1280 totalling 1.2KiB

2021-10-21 21:59:06.578387: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 97536 totalling 95.2KiB

2021-10-21 21:59:06.578520: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 100096 totalling 97.8KiB

2021-10-21 21:59:06.578655: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 180224 totalling 352.0KiB

2021-10-21 21:59:06.578787: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 900096 totalling 879.0KiB

2021-10-21 21:59:06.578912: I tensorflow/core/common_runtime/bfc_allocator.cc:1072] Sum Total of in-use chunks: 1.40MiB

2021-10-21 21:59:06.579036: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] total_region_allocated_bytes_: 4183621632 memory_limit_: 4183621632 available bytes: 0 curr_region_allocation_bytes_: 8367243264

2021-10-21 21:59:06.579256: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] Stats:

Limit: 4183621632

InUse: 1465600

MaxInUse: 1465600

NumAllocs: 157535

MaxAllocSize: 900096

Reserved: 0

PeakReserved: 0

LargestFreeBlock: 0

2021-10-21 21:59:06.579654: W tensorflow/core/common_runtime/bfc_allocator.cc:468] *___________________________________________________________________________________________________

2021-10-21 21:59:06.580231: W tensorflow/core/framework/op_kernel.cc:1680] Resource exhausted: failed to allocate memory

Traceback (most recent call last):

File "D:\works\python\TensorFlows\tensorflows.py", line 107, in <module>

optimizer.apply_gradients(zip(grads, [W, B]))

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1367, in binary_op_wrappe

return func(x, y, name=name)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrappe

return target(*args, **kwargs)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\math_ops.py", line 548, in subtract

return gen_math_ops.sub(x, y, name)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 10643, in sub

_ops.raise_from_not_ok_status(e, name)

File "D:\IDE\python\310\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status

six.raise_from(core._status_to_exception(e.code, message), None)

File "<string>", line 3, in raise_from

tensorflow.python.framework.errors_impl.ResourceExhaustedError: failed to allocate memory [Op:Sub]

进程完成，退出码 1

请问有什么解决办法？

tensorflow

c++

python

深度学习

http

回答

和开发者交流更多问题细节吧，去写回答

相似问题

问使用tensorflow进行迭代过程出现failed to allocate memory的bug？

回答

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐