tesla t4的坑Unable to load the kernel module 'nvidia.ko'.ipynb
eventlet在ubuntu上出现OSError protocol not found

tensorflow资源耗净 Resource exhausted OOM when allocating tensor with shape

妙音 posted @ 2020年5月01日 13:02 in tensorflow with tags tensorflow nvidia , 6385 阅读

描述

 
tensorflow跑训练集经常会遇到错误Resource exhausted: OOM when allocating tensor with shape[64,33,33,2048]
 

错误内容

tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[64,33,33,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node SecondStageBoxPredictor_1/ResizeBilinear (defined at /app/models/research/object_detection/predictors/heads/mask_head.py:149) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[[total_loss/_7771]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[64,33,33,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node SecondStageBoxPredictor_1/ResizeBilinear (defined at /app/models/research/object_detection/predictors/heads/mask_head.py:149) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node SecondStageBoxPredictor_1/ResizeBilinear:
 SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu (defined at /app/models/research/slim/nets/resnet_v1.py:136)

Input Source operations connected to node SecondStageBoxPredictor_1/ResizeBilinear:
 SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu (defined at /app/models/research/slim/nets/resnet_v1.py:136)
 

原因

 
tensorflow在为张量shape[64,33,33,2048]分配gpu内存是发现资源不够。
 
假如数据类型是int8, 该张量需要的内存大小64*33*33*2048*1B=142737408B = 142.7MB
 

解决方法

 
1. 降低图片质量
2. batch_size改成1
3. 改用大内存的显卡
4. 增加显卡, 并行训练
 

 

Avatar_small
Micah 说:
2020年11月13日 15:41

This is just the information I am finding everywhere. Thanks for your blog, I just subscribe your blog. This is a nice blog..   먹튀검증

Avatar_small
Digital_Zone 说:
2020年11月16日 17:19 I’m impressed, I have to admit. Genuinely rarely should i encounter a blog that’s both educative and entertaining, and without a doubt, you’ve got hit the nail for the head. Your thought is outstanding; the problem is a thing that there are not enough everyone is speaking intelligently about. My business is happy i found this in my hunt for some thing regarding this. Seo
Avatar_small
Digital_Zone 说:
2020年11月16日 18:14

Wow, What a Excellent post. I really found this to much informatics. It is what i was searching for.I would like to suggest you that please keep sharing such type of info.Thanks Webdesign

Avatar_small
Digital_Zone 说:
2020年11月16日 18:17

Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. Seo

Avatar_small
Digital_Zone 说:
2020年11月16日 18:20

I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. Lead generation

Avatar_small
Digital_Zone 说:
2020年11月16日 18:30

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. Lead generation

Avatar_small
Digital_Zone 说:
2020年11月16日 18:34

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks Leadgeneratie

Avatar_small
Digital_Zone 说:
2020年11月16日 18:37

I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. Seo

Avatar_small
Digital_Zone 说:
2020年11月16日 18:40

Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from you. Seo

Avatar_small
Digital_Zone 说:
2020年11月16日 18:44

I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me.. Thanks for all your help and wishing you all the success in your business. Leads genereren

Avatar_small
Digital_Zone 说:
2020年11月16日 18:47

This is highly informatics, crisp and clear. I think that everything has been described in systematic manner so that reader could get maximum information and learn many things. Seo optimalisatie

Avatar_small
Digital_Zone 说:
2020年11月16日 18:50

I know your expertise on this. I must say we should have an online discussion on this. Writing only comments will close the discussion straight away! And will restrict the benefits from this information. Seo no cure no pay

Avatar_small
Digital_Zone 说:
2020年11月16日 18:54

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me. Lead generation

Avatar_small
Digital_Zone 说:
2020年11月16日 18:57

I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. Seo leads

Avatar_small
Digital_Zone 说:
2020年11月16日 19:03

Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. Webdesign

Avatar_small
Digital_Zone 说:
2020年11月16日 19:06

This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post. I will visit your blog regularly for Some latest post. Seo leadgeneration

Avatar_small
Digital_Zone 说:
2020年11月16日 19:10

Please let me know if you’re looking for a article writer for your site. You have some really great posts and I feel I would be a good asset. If you ever want to take some of the load off, I’d absolutely love to write some material for your blog in exchange for a link back to mine. Please send me an email if interested. Thank you! Reiki ontspanningsmassage

Avatar_small
Digital_Zone 说:
2020年11月16日 19:15

Thanks for every other informative site. The place else may just I get that kind of information written in such an ideal means? I have a venture that I’m just now operating on, and I have been on the look out for such information. Seo leads

Avatar_small
Digital_Zone 说:
2020年11月16日 19:19

You have a real ability for writing unique content. I like how you think and the way you represent your views in this article. I agree with your way of thinking. Thank you for sharing. Lead generation

Avatar_small
Digital_Zone 说:
2020年11月16日 19:23

I’ve been surfing online more than three hours today, yet I never found any interesting article like yours. It’s pretty worth enough for me. In my opinion, if all webmasters and bloggers made good content as you did, the web will be a lot more useful than ever before. Webdesign

Avatar_small
Digital_Zone 说:
2020年11月16日 19:25

Impressive web site, Distinguished feedback that I can tackle. Im moving forward and may apply to my current job as a pet sitter, which is very enjoyable, but I need to additional expand. Regards. Webdesign

Avatar_small
Timo Zlatko 说:
2020年12月14日 23:28

Central Bank of India offers CBI net banking services to all the account holders over various branches, and these services have made banking faster, easier and more convenient than ever before, the customer of CBI net banking online allows one to carry out a range of financial as well as non-financial transactions online, and CBI internet banking login allows users to go through many banking operations securely from their homes. KMB customer care

Avatar_small
Digital_Zone 说:
2020年12月18日 04:56

Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. Crawley Accountants

Avatar_small
Digital_Zone 说:
2020年12月18日 04:59

Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too. There tend to be not many people who can certainly write not so simple posts that artistically. Continue the nice writing East Grinstead Accountants

Avatar_small
Model Paper 2021 说:
2020年12月23日 13:45

I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post. <a href="https://www.edpost.in/">Model Paper 2021</a>

Avatar_small
Pak24tv 说:
2020年12月26日 13:27

I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. top 10 tallest buildings in the world what is vpn periodic table voided check

Avatar_small
Pak24tv 说:
2020年12月26日 13:42

I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. jazz daily call packages jazz monthly sms packages jazz weekly internet packages how to load jazz card

Avatar_small
Edward Bartoletti 说:
2020年12月29日 17:40

MSBSHSE SSC Examination every year more than laks of students So, the candidates can practice the Maharashtra 10th Model Paper 2021 & Sample Questions Paper. Applicants need to complete the given by placing a correct Previous Model Paper in it. Competitors who are having the necessary knowledge The Candidates can know all the points by practicing Maharashtra SSC Examination the provided Sample Paper 2021 regarding General English. The definite the Maharashtra 10th Bit Bank, Question Bank and pervious Paper, Model Paper 2021 is used when the referent of the noun phrase is assumed to be unique or known from the sample paper For more information on Paper usage, see the Sample Paper 2021 Given Below inter Model Paper 2021. Balbharti 10th Class Model Paper 2021

Avatar_small
Legend Seo 说:
2021年1月11日 22:34

I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. 메이저사이트

Avatar_small
Imamia Quran Academy 说:
2023年7月27日 06:01

The contents of the article are very original. thanks for the work, may be a inspires for everything. I am from Imamia Quran Academy. These online Shia Quran Courses of Shia Quran Academy include basic learning of Yassarnal Quran which will make it easy for them to learn Quran and recite Holy Quran with correct Tajweed.

Avatar_small
qk 说:
2023年8月22日 23:12

If you are looking for more information about flat rate locksmith Las Vegas check that right away. VISA PARA SA CANADA

Avatar_small
qk 说:
2023年8月24日 01:27 Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. ufabet เว็บหลัก
Avatar_small
qk 说:
2023年8月24日 22:16

It’s appropriate time to make some plans for the future and it is time to be happy. I have read this post and if I could I wish to suggest you few interesting things or advice. Perhaps you could write next articles referring to this article. I desire to read even more things about it! ufabet เว็บแม่

Avatar_small
Johny 说:
2023年8月27日 01:41

Excellent .. Amazing .. I’ll bookmark your blog and take the feeds also…I’m happy to find so many useful info here in the post, we need work out more techniques in this regard, thanks for sharing. best vibrating dildo for women

Avatar_small
qk 说:
2023年8月27日 21:12

I would like to say that this blog really convinced me to do it! Thanks, very good post. <a href="https://vp-academic-affairs.kau.edu.sa/Default-0004963-AR">المواقع السعودية</a>

Avatar_small
qk 说:
2023年9月02日 23:55

You know your projects stand out of the herd. There is something special about them. It seems to me all of them are really brilliant! 안전놀이터

Avatar_small
qk 说:
2023年9月04日 22:23

I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. 越南商业签证

Avatar_small
qk 说:
2023年9月06日 13:19

This blog is so nice to me. I will keep on coming here again and again. Visit my link as well.. djarumtoto

Avatar_small
qk 说:
2023年9月06日 20:26

thanks for the tips and information..i really appreciate it.. <a href="https://www.ufabetwins.info/แทงบอลออนไลน์/">แทงบอลสเต็ป</a>

Avatar_small
qk 说:
2023年9月06日 20:27

thanks for the tips and information..i really appreciate it.. แทงบอลสเต็ป

Avatar_small
qk 说:
2023年9月06日 20:46

Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!.. เว็บบอลออนไลน์

Avatar_small
alex 说:
2023年9月07日 18:32

A very excellent blog post. I am thankful for your blog post. I have found a lot of approaches after visiting your post. ufabet เว็บตรงไม่ผ่านเอเย่นต์

Avatar_small
qk 说:
2023年9月09日 15:55

Cool stuff you have got and you keep update all of us. plastic surgeon

Avatar_small
qk 说:
2023年9月09日 16:23

Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. AVALIADOR PREMIADO

Avatar_small
monthly cleaning ser 说:
2023年9月11日 20:00

There is probably not a more fulfilling feeling as compared to coming home with a spotless residence, particularly when another individual did every bit of the task; there just isn't anything left to accomplish but set down in the hammock inside the sun! With all the hectic lifestyle a lot of people have and also both lovers working, lots of people find they don't really have much time left regarding domestic servicing. Before you browse the Web or call up somewhere to try to get a maid.

Avatar_small
qk 说:
2023年9月17日 17:37

Keep up the good work , I read few posts on this web site and I conceive that your blog is very interesting and has sets of fantastic information. visa eta canada desde austria

Avatar_small
qk 说:
2023年9月19日 01:44

I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. auto wild mahjong

Avatar_small
qk 说:
2023年9月22日 19:36

Thank you because you have been willing to share information with us. we will always appreciate all you have done here because I know you are very concerned with our. Jogo do Tigre

Avatar_small
alex 说:
2023年9月23日 17:07

Thank you so much as you have been willing to share information with us. We will forever admire all you have done here because you have made my work as easy as ABC. กีฬาไอซ์สเก็ต

Avatar_small
qk 说:
2023年9月27日 00:08

I really like your writing style, great information, thankyou for posting. chatgpt app ios

Avatar_small
qk 说:
2023年10月08日 14:41

Great survey, I'm sure you're getting a great response. factsscope

Avatar_small
qk 说:
2023年10月09日 21:20

Thank you very much for writing such an interesting article on this topic. This has really made me think and I hope to read more. taruhan olahraga

Avatar_small
qk 说:
2023年10月11日 22:26

Thanks for providing recent updates regarding the concern, I look forward to read more. Fortune Tiger

Avatar_small
qk 说:
2023年10月11日 23:05

I really like your take on the issue. I now have a clear idea on what this matter is all about.. NEOTONICS

Avatar_small
qk 说:
2023年10月17日 01:57 I love the way you write and share your niche! Very interesting and different! Keep it coming! AVALIADOR PREMIADO
Avatar_small
qk 说:
2023年10月17日 01:58

I love the way you write and share your niche! Very interesting and different! Keep it coming! AVALIADOR PREMIADO

Avatar_small
qk 说:
2023年10月22日 00:09

Merely a smiling visitant here to share the love (:, btw outstanding style. บาคาร่าเว็บไหนดี2022

Avatar_small
qk 说:
2023年11月01日 17:01

I read a article under the same title some time ago, but this articles quality is much, much better. How you do this.. JOGO DO TIGRE

Avatar_small
qk 说:
2023年11月01日 19:30

I havent any word to appreciate this post.....Really i am impressed from this post....the person who create this post it was a great human..thanks for shared this with us. FORTUNE TIGER

Avatar_small
qk 说:
2023年11月01日 19:45

I think this is a really good article. You make this information interesting and engaging. You give readers a lot to think about and I appreciate that kind of writing. CODIGOS LUCRATIVOS

Avatar_small
qk 说:
2023年12月25日 01:00

I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Jago168 login

Avatar_small
qk 说:
2023年12月25日 01:48

I read your blog frequently and I just thought I’d say keep up the amazing work! Lektoto

Avatar_small
qk 说:
2023年12月25日 01:48

I read your blog frequently and I just thought I’d say keep up the amazing work! Lektoto

Avatar_small
qk 说:
2023年12月25日 02:23

I admire what you have done here. I like the part where you say you are doing this to give back but I would assume by all the comments that this is working for you as well. pasar123 login

Avatar_small
qk 说:
2023年12月28日 03:54

Hi there! Nice post! Please tell us when I will see a follow up! respin123 login

Avatar_small
qk 说:
2024年2月16日 00:42

I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. slot scbet88

Avatar_small
zlinrary 说:
2024年4月02日 23:28

"In the library, every book is a doorway to a different world, a portal to a new reality waiting to be explored."


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter