Scrapy的Scrapyd调度蜘蛛太慢了
Scrapy的Scrapyd调度蜘蛛太慢了
提问于 2012-02-06 22:34:23
我正在运行Scrapyd，在同时启动4个爬虫时遇到了一个奇怪的问题。
2012-02-06 15:27:17+0100 [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [06/Feb/2012:14:27:16 +0000] "POST /schedule.json HTTP/1.1" 200 62 "-" "python-requests/0.10.1"
2012-02-06 15:27:17+0100 [HTTPChannel,1,127.0.0.1] 127.0.0.1 - - [06/Feb/2012:14:27:16 +0000] "POST /schedule.json HTTP/1.1" 200 62 "-" "python-requests/0.10.1"
2012-02-06 15:27:17+0100 [HTTPChannel,2,127.0.0.1] 127.0.0.1 - - [06/Feb/2012:14:27:16 +0000] "POST /schedule.json HTTP/1.1" 200 62 "-" "python-requests/0.10.1"
2012-02-06 15:27:17+0100 [HTTPChannel,3,127.0.0.1] 127.0.0.1 - - [06/Feb/2012:14:27:16 +0000] "POST /schedule.json HTTP/1.1" 200 62 "-" "python-requests/0.10.1"
2012-02-06 15:27:18+0100 [Launcher] Process started: project='thz' spider='spider_1' job='abb6b62650ce11e19123c8bcc8cc6233' pid=2545
2012-02-06 15:27:19+0100 [Launcher] Process finished: project='thz' spider='spider_1' job='abb6b62650ce11e19123c8bcc8cc6233' pid=2545
2012-02-06 15:27:23+0100 [Launcher] Process started: project='thz' spider='spider_2' job='abb72f8e50ce11e19123c8bcc8cc6233' pid=2546
2012-02-06 15:27:24+0100 [Launcher] Process finished: project='thz' spider='spider_2' job='abb72f8e50ce11e19123c8bcc8cc6233' pid=2546
2012-02-06 15:27:28+0100 [Launcher] Process started: project='thz' spider='spider_3' job='abb76f6250ce11e19123c8bcc8cc6233' pid=2547
2012-02-06 15:27:29+0100 [Launcher] Process finished: project='thz' spider='spider_3' job='abb76f6250ce11e19123c8bcc8cc6233' pid=2547
2012-02-06 15:27:33+0100 [Launcher] Process started: project='thz' spider='spider_4' job='abb7bb8e50ce11e19123c8bcc8cc6233' pid=2549
2012-02-06 15:27:35+0100 [Launcher] Process finished: project='thz' spider='spider_4' job='abb7bb8e50ce11e19123c8bcc8cc6233' pid=2549 我已经有了Scrapyd的这些设置：
[scrapyd]
max_proc = 10为什么Scrapyd没有在同一时间运行蜘蛛，就像他们计划的那样快？
回答 2
Stack Overflow用户
回答已采纳
发布于 2012-02-14 00:03:17
我已经通过编辑30行的scrapyd/app.py解决了这个问题。
将timer = TimerService(5, poller.poll)更改为timer = TimerService(0.1, poller.poll)
编辑:下面由AliBZ提供的关于配置设置的注释是更改轮询频率的更好方法。
Stack Overflow用户
发布于 2012-02-07 02:11:59
从我使用scrapyd的经验来看，它不会像你计划的那样立即运行一个爬虫。它通常会等待一小段时间，直到当前爬行器启动并运行，然后启动下一个爬行器进程(scrapy crawl)。
因此，scrapyd逐个启动进程，直到达到max_proc计数。
从你的日志中，我看到你的每个蜘蛛都在运行大约1秒。我想，如果你的蜘蛛至少跑30秒，你会看到所有的蜘蛛都在跑。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/9161724
复制相关文章
相似问题
腾讯云开发者
