使用从网站提取的数据创建数据帧
使用从网站提取的数据创建数据帧
提问于 2021-06-24 08:57:06
我正在尝试使用selenium从Facebook部分获取分享数量，如下所示：
from selenium import webdriver
import time
chrome_options = webdriver.ChromeOptions()
total = []
shares = []
comments = []
reactions = []
query=["www.stackoverflow.com","www.facebook.com"]
driver = webdriver.Chrome(mypath,chrome_options=chrome_options)
driver.get('https://www.sharedcount.com/')
textarea = driver.find_element_by_xpath('//textarea')
for x in query:
textarea.send_keys(query)
button = driver.find_element_by_xpath("//button[@class='button button_accent-green']")
button.click()
body = driver.find_element_by_xpath("//tb-data tb-data_face[@class='bold-text']")
total.append(body)
sharing=driver.find_element_by_xpath("//tb-line-info__text tb-line-info__text_left tb-line-info__face[@class='bold-text']")
shares.append(sharing)
com=driver.find_element_by_xpath("//tb-line-info__text tb-line-info__text_left[@class='bold-text']")
comments.append(com)利用查询列表中每个url的信息创建新的数据帧。然而，我得到了这个错误
InvalidSelectorException: Message: invalid selector: Unable to locate an element with the xpath expression //tb-data tb-data_face[@class='bold-text'] because of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string '//tb-data tb-data_face[@class='bold-text']' is not a valid XPath expression.
(Session info: chrome=91.0.4472.114)有没有人知道如何修复错误并得到以下输出？
url shares comments reactions
www.stackoverflow.com 11.2k 3.8k 9.1k
www.facebook.com 1920.4m 64.9m 517.7m 在使用WebDriverWait之后，我得到了另一个错误：
24 button.click()
---> 25 WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, '//td[@class="tb-data tb-data-face"]/div[@class="bold-text"]')))
26 body = driver.find_element_by_xpath('//td[@class="tb-data tb-data-face"]/div[@class="bold-text"]')
27 total.append(body)
~/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/support/wait.py in until(self, method, message)
78 if time.time() > end_time:
79 break
---> 80 raise TimeoutException(message, screen, stacktrace)
81
82 def until_not(self, method, message=''):
TimeoutException: Message: 回答 1
Stack Overflow用户
回答已采纳
发布于 2021-06-24 10:06:15
我相信您在XPath中遗漏了<td>和<div>标记。它应该是
'//td[@class="tb-data tb-data_face"]/div[@class="bold-text"]'
您还需要使用相同的思路调整sharing和com XPaths。
编辑1:尝试等待图元的可见性：
导入：
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import Byfor x in query:
...
button.click()
# Wait 10 seconds for the element to be visible
WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, '//td[@class="tb-data tb-data_face"]/div[@class="bold-text"]')))
body = driver.find_element_by_xpath('//td[@class="tb-data tb-data_face"]/div[@class="bold-text"]')
total.append(body)
...页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/68108424
复制相关文章
相似问题
腾讯云开发者
