Python phantomjs加载网页不正确[英] Python phantomjs loading webpage not correct

问题描述

我有一个从这个链接提取的问题

http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&company=5250

从这个链接给我带来数据,而不是主页本身.http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=all

知道为什么会这样吗?我正在使用 PhantomJS 硒和美丽的汤来帮助我.

# The standard library modules
import os
import sys
import re
import sqlite3
import locale
# The wget module
import wget
import time
import calendar
from datetime import datetime
# The BeautifulSoup module
from bs4 import BeautifulSoup

# The selenium module
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


def getURLS(url):
    driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true'])
    driver.get(url) # load the web page
    src = driver.page_source
    #Get text and split it
    soup = BeautifulSoup(src, 'html5lib')

    print soup

link ='http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&company=5250'
getURLS(link)

Alex Lucaci 的解决方案

def getURLS(url):
    driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true'])
    driver.get(url) # load the web page
    src = driver.page_source
    category_select = Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]'))
    category_select.select_by_visible_text("Financial Results")
    category_select2 = Select(driver.find_element_by_xpath('//*[@id="bm_sub_announcement_types"]'))
    category_select2.select_by_visible_text("Financial Results")
    category_select3 = Select(driver.find_element_by_xpath('//*[@id="bm_company_list"]'))
    category_select3.select_by_visible_text("7-ELEVEN MALAYSIA HOLDINGS BERHAD (5250)")
    driver.find_element_by_xpath('//*[@id="bm_company_announcements_search_form"]/input[1]').click()
    src = driver.page_source
    soup = BeautifulSoup(src, 'html5lib')
    link="http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=all"
    getURLS(link)

推荐答案

当您保存源时,页面未完全加载您提交的帖子,因此请尝试等待几秒钟,然后再获取页面源:

p>
def getURLS(url):
driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true'])
driver.get(url) # load the web page
time.sleep(5)# waiting for 5 seconds before fetching the source
src = driver.page_source
#Get text and split it
soup = BeautifulSoup(src, 'html5lib')

print soup

要执行下拉选择,您必须导入 Select 类,如下所示:from selenium.webdriver.support.ui import Select 然后您必须像这样选择下拉元素:

category_select = Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]'))
category_select.select_by_visible_text('Financial Results')

在我的示例中,我已经为 -Category- 下拉菜单完成了此操作,请按照每个类别的确切步骤进行操作.请注意,通过 xpath 选择下拉菜单是最好的方法,您可以通过使用 Google Chrome -> 右键单击​​元素 -> Inspect-> 右键单击​​出现的右侧菜单中的 <select> -> Copy -> 来实现此目的复制 Xpath

当您选择了所有元素后,您必须单击提交并等待几秒钟以加载,然后您将获取源代码.

如果我的回答对你有帮助,请告诉我.

本文地址:https://www.itbaoku.cn/post/1740202.html