blocks|key|5822733|text|正如您在评论中提到的，您使用的是ext3。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|5822734|众所周知，rm对于ext3上的大型文件的性能很差；这是ext4中修复的问题之一。例如，请参见这个职位或核菜鸟+(其中提到区段可以提高大文件的删除和截断速度)。|5822735|我不知道这在多大程度上适用于您典型的文件大小。我希望它至少能应用一点，因为在大约200+at的时间里，您已经在ext3上使用间接块，而在ext4上可能只使用一个区段。|5822736|作为一种解决办法(因为您可能不会仅仅为了这一点而升级到ext4+)，每次只删除几个文件，并在删除之间添加一个sleep。这不是很漂亮，但应该有助于减轻负荷。|5822737|另外，如果丢失断电文件不是问题(因为它是某种缓存)，那么您可以将它们放在一个单独的分区中，在引导时再次使用mkfs，并使用没有日志甚至ext2的ext3。造成高负载的原因可能是日志被刷新到磁盘，与读取冲突(您在另一篇文章中提到，您有很多并发读取)。|entityMap|0|LINK|mutability|MUTABLE|url|http://www.phoronix.com/forums/showpost.php?p=80150&postcount=25|1|http://kernelnewbies.org/Ext4^0|G|4|0|5|2|R|4|1A|4|0|1F|3|1|0|1J|4|1W|4|0|R|4|1I|5|0|1H|4|1V|4|20|4^^$0|@$1|2|3|4|5|6|7|W|8|@$9|X|A|Y|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|Z|8|@$9|10|A|11|B|C]|$9|12|A|13|B|C]]|D|@$9|14|A|15|1|16]|$9|17|A|18|1|19]]|E|$]]|$1|H|3|I|5|6|7|1A|8|@$9|1B|A|1C|B|C]|$9|1D|A|1E|B|C]]|D|@]|E|$]]|$1|J|3|K|5|6|7|1F|8|@$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]]|D|@]|E|$]]|$1|L|3|M|5|6|7|1K|8|@$9|1L|A|1M|B|C]|$9|1N|A|1O|B|C]|$9|1P|A|1Q|B|C]]|D|@]|E|$]]]|N|$O|$5|P|Q|R|E|$S|T]]|U|$5|P|Q|R|E|$S|V]]]]

As you mentioned in a comment, you are using <code>ext3</code>.

It is well known that <code>rm</code> performance for large files on ext3 is poor; it is one of the things which were fixed in <code>ext4</code>. See for instance <a href="http://www.phoronix.com/forums/showpost.php?p=80150&amp;postcount=25" rel="nofollow noreferrer">this post</a>, or <a href="http://kernelnewbies.org/Ext4" rel="nofollow noreferrer">kernelnewbies</a> (which mentions extents improve delete and truncate speeds for large files).

I do not know how much that applies to your typical file sizes. I would it expect it to apply at least a little, since with around 200kB you would already be using indirect blocks on <code>ext3</code>, versus possibly a single extent on <code>ext4</code>.

<hr>

As a workaround (since you probably will not upgrade to <code>ext4</code> just for that), delete only a few files each time and add a <code>sleep</code> between the deletions. It is not pretty, but should help reducing the load.

Also, if losing the files on power loss is not a problem (since it is a cache of some sort), you could put them in a separate partition which you <code>mkfs</code> again on boot, and use <code>ext3</code> without a journal or even <code>ext2</code>. The cause for the high load is probably the journal being flushed to disk conflicting with the reads (you mentioned in another post that you have lots of concurrent reads).

blocks|key|3824064|text|也许壳是问题的原因。您应该直接使用find：find+/dir+-mindepth+1+-maxdepth+1+-delete|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|entityMap^0|M|15^^$0|@$1|2|3|4|5|6|7|G|8|@$9|H|A|I|B|C]]|D|@]|E|$]]]|F|$]]

Maybe the shell is the cause of the problem. You should use directly find : <code>find /dir -mindepth 1 -maxdepth 1 -delete</code>

blocks|key|5822574|text|那只是25万个左右的文件，应该不是一个真正的问题--您使用的是什么文件系统，这个卷还用于其他什么吗？|type|unstyled|depth|inlineStyleRanges|entityRanges|data|entityMap^0^^$0|@$1|2|3|4|5|6|7|C|8|@]|9|@]|A|$]]]|B|$]]

That's only 250,000 files or so, shouldn't be a problem really - what file system are you using and is this volume used for anything else?

blocks|key|3824420|text|当您有许多您所描述的文件时，您每次都要调用该命令。此外，您必须记住日志FS，您正在处理缓冲区命中和元数据，这会对进程时间产生很大影响。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3824421|最好的方法是像上面提到的那样使用find命令，只使用一个不太明显的特性。|3824422|find+/+-name+filename.*+-exec+/bin/rm+-f+'{}'+\%2B|code-block|syntax|javascript|3824423|基本上"%2B“是你的朋友。这样做是在集合中创建文件名，并每组调用一次rm命令。这与'xargs‘所做的几乎是一样的，但是如果在BSD/Linux上，则不必担心正确的标志。|3824424|很好奇这对你来说有多快。所以如果你还在的话请回复。祝你好运！|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|P|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|Q|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|R|8|@]|9|@]|A|$]]]|M|$]]

When you have a lot of files like you described you are invoking the command each time. Also you have to keep in mind on journaled FS you're dealing with buffer hits and metadata which can greatly impact process times.

Your best bet is to use the find command like mentioned above only with a less than obvious feature. 

<pre><code>find / -name filename.* -exec /bin/rm -f '{}' \+
</code></pre>

Basically the "+" is your friend. What this does is create the filenames in sets and invokes rm command once per set. That's pretty much the same as what 'xargs' does but you don't have to worry about the correct flags if on BSD/Linux. 

Very curious how much that speeds it up for you. So reply back if you are still around. Good luck !

blocks|key|5823388|text|在研究了文件系统基准之后，我选择JFS作为mythtv视频文件的文件存储库，因为文件删除很快(+mythtv等待删除完成，这使IO变得迟缓)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5823389|您还可以通过“查找”和“xargs”调用“rm”，而不是rm+-rf。这可能会更快：|5823390|find+<dir>+%7C+xargs+rm|code-block|syntax|javascript|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|L|8|@]|9|@]|A|$G|H]]]|I|$]]

After studying filesystem benchmarks I've choosen JFS as file store for my mythtv video files because file deletes are fast (and mythtv waits for the delete to finish, making the IO sluggish).

You could also invoke 'rm' through 'find' and 'xargs' instead of rm -rf. This might be faster:

<pre><code>find &lt;dir&gt; | xargs rm
</code></pre>

blocks|key|5823437|text|如何将这些文件的列表连接到Perl并使用其unlink函数？|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5823438|find+<dir>+%7C+perl+-nle+'unlink;'|code-block|syntax|javascript|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|I|8|@]|9|@]|A|$E|F]]]|G|$]]

How about piping a list of those files to Perl and using its unlink function?

<pre><code>find &lt;dir&gt; | perl -nle 'unlink;'
</code></pre>

blocks|key|3824378|text|我同意它不应该花那么长的时间，但取决于所使用的底层存储，它可能是预期的密集读取。我认为最终您最好的解决方案是添加更多的磁盘，并将您的活动分散在其中。如果您沿着这条路线走，RAID在某些情况下可能会有所帮助。在这段时间里iostat跟你说了什么？您还可以使用for循环，并在'time‘中包装'rm’命令以获得一些附加信息。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3824379|另一种可能性，当然取决于您的设置和应用程序，但也许您可以将这些缓存文件放在不同的分区上，并定期格式化驱动器，而不是删除这些文件？我认为运行mkfs会大大减少时间，但是在这种情况下，您的应用程序将不可用，因此不太理想。|3824380|我也喜欢更经常地清洗它们的想法。在cron里说，你每小时都安排这样的事情：|3824381|查找./+-maxdepth+1+-type+f+-name“一些模式”-ctime+%2B1+-exec+rm+-f+{}；|3824382|这将删除每一个早于24小时的文件，而不是一次全部删除。|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|M|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|N|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|O|8|@]|9|@]|A|$]]]|J|$]]

I agree that it shouldn't take that long, but depending on the underlying storage being used, it may be expected with the intensive reads. I think ultimately your best solution is to add additional disks and split up your activities amongst them. RAID could help in some scenarios if you go down that route. What does iostat tell you during these times? You could also use a for loop and wrap the 'rm' commands in 'time' to get some additional information. 

Another possibility, depending on your setup and the application of course, but maybe you could make these cache files on a different partition and just format the drive periodically rather than remove the files? I think running mkfs would cut the time down considerably, but your application is going to be unavailable while that happens so it won't be ideal. 

I also like the idea of cleaning them up more often. Say in cron you schedule something like this every hour:

find ./ -maxdepth 1 -type f -name "some pattern" -ctime +1 -exec rm -f {} \;

This would delete every file that is older than 24 hours, rather then trying to do them all at once.

I generate more than 50GB of cache files on my RHEL server (and typical file size is 200kb so no of files is huge). When I try to delete these files it takes 8-10 hours.

However, the bigger issue is that the system load goes to critical for these 8-10 hours. Is there anyway where I can keep the system load under control during the deletion.

I tried using

<pre><code>nice -n19 rm -rf *
</code></pre>

but that doesn't help in system load.

P.S. I asked the same question on <a href="https://superuser.com/questions/124707/deleting-large-no-of-files-on-linux-eats-up-cpu">superuser.com</a> but didn't get a good enough answer so trying here.

Deleting large no of files on linux eats up CPU

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我在我的RHEL服务器上生成了超过50 so的缓存文件(典型的文件大小是200 so，所以没有一个文件是巨大的)。当我试图删除这些文件时，需要8-10小时。然而，更大的问题是，在这8-10个小时内，系统负载达到了临界状态。在删除过程中，我是否可以控制系统负载。我试着用nice -n19 rm -rf *但这对系统负载没有帮助。我在superuser.com上问了同样的问题，但没有得到足够好的答案，

问在linux上删除大的no文件占用了CPU。
EN

回答 7

Server Fault用户

Server Fault用户

Server Fault用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在linux上删除大的no文件占用了CPU。EN

回答 7

Server Fault用户

Server Fault用户

Server Fault用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在linux上删除大的no文件占用了CPU。
EN