下面的代码是谷歌趋势爬虫,它使用了来自"https://github.com/GeneralMills/pytrends“的非官方应用程序接口。我的代码运行良好,但一个问题是,没有人知道谷歌趋势爬虫的限制。因此,如果我运行包含2000个或更多"DNA“列表的Crawler,那么我会错误地说我已经超过了请求限制。如果我超过了限制,所有在限制之前爬取的数据都将丢失,因为我是在代码末尾写入csv的。有没有一种方法可以将我的数据写入csv for每个循环,所以即使我超过了限制,至少我在达到限制之前有数据?谢谢
from pytrends.request import TrendReq
from datetime import datetime
import pandas as pd
import time
import xlsxwriter
pytrends = TrendReq(hl='en-US,tz=360')
Data = pd.DataFrame()
#for loop check writer path
path = "C:/Users/aijhshin/Workk/GoogleTrendCounter.txt"
#file = open(path,"a")
#setting index using 'apple' keyword
kw_list = ['apple']
pytrends.build_payload(kw_list, cat=0, timeframe='today 5-y', geo='', gprop='')
Googledate = pd.DataFrame(pytrends.interest_over_time())
Data['Date'] = Googledate.index
#Google Trend Crawler limit = 1600 request per day
for i in range(len(DNA)):
kw_list = [DNA[i]]
pytrends.build_payload(kw_list, cat=0, timeframe='today 5-y', geo='', gprop='')
#results
df = pd.DataFrame(pytrends.interest_over_time())
if(df.empty == True):
Data[DNA[i]] = ""
else:
df.index.name = 'Date'
df.reset_index(inplace=True)
Data[DNA[i]] = df.loc[:, DNA[i]]
#test for loop process
file = open(path,"a")
file.write(str(i) + " " + str(datetime.now()) + " ")
file.write(DNA[i] +'\n')
file.close()
#run one per nine second (optional)
#time.sleep(9)
#writing csv file (overwrite each time)
Data.to_csv('Google Trend.csv')
print("Crawling Done")https://stackoverflow.com/questions/47625436
复制相似问题