blocks|key|3699153|text|这是做不到的。没有官方的robots.txt标准，这实际上只是一个惯例，不同的网络爬虫正在努力尊重和正确解释。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3699154|但是Googlebot支持通配符，所以您可以有这样的部分：|3699155|User-agent:+Googlebot
Disallow:+/*NAME|code-block|syntax|javascript|3699156|由于大多数网络爬虫无法正确解释通配符，谁也不知道他们是如何解释通配符的，因此可能只为googlebot隔离这条规则是安全的，但我认为，到目前为止，每个大型搜索引擎都可以支持通配符，而谷歌在搜索中所做的一切都已成为事实上的标准。|entityMap|0|LINK|mutability|MUTABLE|url|http://www.sitepoint.com/blogs/2005/10/21/googles-hidden-protocol/^0|C|A|0|2|E|0|0|0^^$0|@$1|2|3|4|5|6|7|V|8|@$9|W|A|X|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|Y|8|@]|D|@$9|Z|A|10|1|11]]|E|$]]|$1|H|3|I|5|J|7|12|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|13|8|@]|D|@]|E|$]]]|O|$P|$5|Q|R|S|E|$T|U]]]]

it cannot be done. there is no official standard for <code>robots.txt</code>, it's really just a convention that various web-crawlers are trying to respect and correctly interpret.

However <a href="http://www.sitepoint.com/blogs/2005/10/21/googles-hidden-protocol/" rel="nofollow noreferrer">Googlebot supports wildcards</a>, so you could have section like this:

<pre><code>User-agent: Googlebot
Disallow: /*NAME
</code></pre>

since most web-crawlers won't interpret wildcards correctly and who knows how they interpret it, it's probably safe to isolate this rule just for googlebot but I would assume that by now every large search engine could support it as well as whatever Google does in search becomes de-facto standard.

blocks|key|3699039|text|++++User-agent:+googlebot
++++Disallow:+/*NAME

++++User-Agent:+slurp
++++Disallow:+/*NAME|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|entityMap^0^^$0|@$1|2|3|4|5|6|7|E|8|@]|9|@]|A|$B|C]]]|D|$]]

<pre><code> User-agent: googlebot
 Disallow: /*NAME

 User-Agent: slurp
 Disallow: /*NAME
</code></pre>

blocks|key|5698416|text|不幸的是，您不能在“不允许”行中幸灾乐祸，因此没有通配符。对于要排除的每个目录，您都需要有一个不允许行。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5698417|User-agent:+*
Disallow:+/A/B/C/NAME/
Disallow:+/D/E/F/NAME/|code-block|syntax|javascript|5698418|这是不幸的，但标准是非常简单的，这是需要这样做的。还请注意，您必须有尾随/在您的不允许行。这是一个相当好的使用robots.txt参考资料。|offset|length|entityMap|0|LINK|mutability|MUTABLE|url|http://www.robotstxt.org/robotstxt.html^0|0|0|1H|G|0^^$0|@$1|2|3|4|5|6|7|R|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|S|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|T|8|@]|9|@$I|U|J|V|1|W]]|A|$]]]|K|$L|$5|M|N|O|A|$P|Q]]]]

You cannot glob in the Disallow line unfortunately, so no wildcards. You would need to have a disallow line for each directory you want to exclude.

<pre><code>User-agent: *
Disallow: /A/B/C/NAME/
Disallow: /D/E/F/NAME/
</code></pre>

It's unfortunate, but the standards is very simplistic and this is how it needs to be done. Also note you must have the trailing / in your disallow line. Here is a fairly good <a href="http://www.robotstxt.org/robotstxt.html" rel="nofollow noreferrer">reference for using robots.txt</a>.

blocks|key|5698218|text|我看到你在堆栈溢出上交叉贴了这个，但我也会把我的答案放在这里。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5698219|不幸的是，您不能在“不允许”行中幸灾乐祸，因此没有通配符。对于要排除的每个目录，您都需要有一个不允许行。|5698220|User-agent:+*
Disallow:+/A/B/C/NAME/
Disallow:+/D/E/F/NAME/|code-block|syntax|javascript|5698221|这是不幸的，但标准是非常简单的，这是需要这样做的。还请注意，您必须有尾随/在您的不允许行。这是一个使用robots.txt的相当好的参考。|offset|length|entityMap|0|LINK|mutability|MUTABLE|url|http://www.robotstxt.org/robotstxt.html^0|0|0|0|1D|J|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|V|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|W|8|@]|9|@$K|X|L|Y|1|Z]]|A|$]]]|M|$N|$5|O|P|Q|A|$R|S]]]]

I see you cross-posted this on Stack Overflow, but I'll put my answer here as well.

You cannot glob in the Disallow line unfortunately, so no wildcards. You would need to have a disallow line for each directory you want to exclude.

<pre><code>User-agent: *
Disallow: /A/B/C/NAME/
Disallow: /D/E/F/NAME/
</code></pre>

It's unfortunate, but the standards is very simplistic and this is how it needs to be done. Also note you must have the trailing / in your disallow line. Here is a <a href="http://www.robotstxt.org/robotstxt.html" rel="nofollow noreferrer">fairly good reference for using robots.txt</a>.

blocks|key|5698253|text|据我所知，没有robots.txt文件解析器支持的模式匹配例程。在这种情况下，您需要用自己的“不允许”语句列出每个文件。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5698254|请记住，在robots.txt文件中列出这些文件将提供这些链接的列表给那些想要向爬虫“隐藏”什么的人，因此如果这是敏感材料，可能会出现安全问题。|5698255|如果这些链接是在由您的服务器提供的HTML中，您还可以向这些链接中添加一个rel="nofollow"到A标记，这将防止大多数爬虫跟踪链接。|offset|length|style|CODE|entityMap^0|0|0|11|E^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|M|8|@$F|N|G|O|H|I]]|9|@]|A|$]]]|J|$]]

To my knowledge there is no pattern matching routine supported by the robots.txt file parsers. In this case you would need to list each of those files with their own Disallow statement.

Keep in mind that listing those files in the robots.txt file will give out a list of those links to anyone who might want to see what you're trying to "hide" from the crawlers, so there may be a security issue if this is sensitive material.

If these links are in HTML served up by your server you can also add a <code>rel="nofollow"</code> to the A tags to those links and it will prevent most crawlers from following the links.

blocks|key|5698325|text|如前所述，robots.txt规范非常简单。然而，我所做的一件事是创建一个简单的名为"robots.txt“的动态脚本(PHP、Python等)，并让它使用脚本的更智能的逻辑巧妙地生成预期的简单结构。您可以遍历子目录，使用正则表达式等。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5698326|您可能需要稍微调整一下您的web服务器，以便它以脚本的形式执行"robots.txt“，而不是仅仅为文件内容提供服务。或者，您可以通过cron作业运行脚本，该作业每晚重新生成一次robots.txt+(无论它需要更新的频率如何)。|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|E|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|F|8|@]|9|@]|A|$]]]|D|$]]

As previously mentioned, the robots.txt spec is pretty simple. However, one thing that I've done is create a dynamic script (PHP, Python, whatever) that's simply named "robots.txt" and have it smartly generate the expected, simple structure using the more intelligent logic of the script. You can walk subdirectories, use regular expressions, etc. 

You might have to tweak your web server a bit so it executes "robots.txt" as a script rather than just serving up the file contents. Alternatively, you can have a script run via a cron job that regenerates your robots.txt once a night (or however often it needs updating)

blocks|key|5698636|text|我在robotstxt.org上看到的最好的文档。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|entityMap|0|LINK|mutability|MUTABLE|url|http://www.robotstxt.org/^0|2|D|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@$A|L|B|M|1|N]]|C|$]]]|D|$E|$5|F|G|H|C|$I|J]]]]

Best documentation I've seen for this is at <a href="http://www.robotstxt.org/" rel="nofollow noreferrer">robotstxt.org</a>.

I have a bunch of files at www.example.com/A/B/C/NAME (A,B,C change around, NAME is static) and I basically want to add a command in robots.txt so crawlers don't follow any such links that have NAME at the end.

What's the best command to use in robots.txt for this?

Robots.txt command

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我在www.example.com/A/B/C/NAME (A，B，C更改，名称是静态的)上有一堆文件，我基本上想在robots.txt中添加一个命令，这样爬虫就不会跟随任何在末尾有名字的链接。在robots.txt中使用的最佳命令是什么？

问Robots.txt命令
EN

回答 7

Server Fault用户

Server Fault用户

Server Fault用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Robots.txt命令EN

回答 7

Server Fault用户

Server Fault用户

Server Fault用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Robots.txt命令
EN