Python Get请求返回不同的HTML,而不是查看源[英] Python get request returning different HTML than view source

问题描述

我正在尝试从我们自己的URL存档中提取粉丝,以便使用NLTK库对其进行一些语言分析.但是,每次尝试从URL刮掉HTML正在返回所有内容,而是粉丝(以及我不需要的评论表格).

首先我尝试使用内置的URLLIB库(和BeautySoup):

import urllib
from bs4 import BeautifulSoup    
html = request.urlopen("http://archiveofourown.org/works/6846694").read()
soup = BeautifulSoup(html,"html.parser")
soup.prettify()
然后我发现关于请求库,以及用户代理程序如何成为问题的一部分,因此我尝试了相同的结果:

import requests
headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36',
        'Content-Type': 'text/html',
}
requests.get("http://archiveofourown.org/works/6846694",headers=headers,timeout=5).text

然后我发现了关于selenium和phantomjs,所以我安装了那些并尝试了这个,而且再次尝试 - 相同的结果:

from selenium import webdriver
from bs4 import BeautifulSoup
browser = webdriver.PhantomJS()
browser.get("http://archiveofourown.org/works/6846694")
soup = BeautifulSoup(browser.page_source, "html.parser")
soup.prettify()

是我在这些尝试中的任何一个尝试中做错了什么,或者这是服务器的问题吗?

推荐答案

如果您需要执行所有JavaScript和所做的Async请求,则最后一个方法是进入正确方向的步骤.你只是错过了一件事 - 你需要给phantomjs time 加载读取源之前的页面(PUP故意).

并且,您还需要单击"继续",即您同意查看成人内容:

from bs4 import BeautifulSoup

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.PhantomJS()
driver.get("http://archiveofourown.org/works/6846694")

wait = WebDriverWait(driver, 10)

# click proceed
proceed = wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Proceed")))
proceed.click()

# wait for the content to be present
wait.until(EC.presence_of_element_located((By.ID, "workskin")))

soup = BeautifulSoup(driver.page_source, "html.parser")
soup.prettify()

其他推荐答案

alexce已经解释了为什么您的代码没有给您想要的原因,如果您想要的只是源中可用的文本,如果添加Param view_adult=true:

import requests
from bs4 import BeautifulSoup
url = "http://archiveofourown.org/works/6846694?view_adult=true"


r= requests.get(url)
soup = BeautifulSoup(r.content, "lxml")
chap = soup.select_one("#chapter-1")
preface = soup.select_one("div.preface.group")


print(preface)
print(chap)

这将为您提供:

<div class="preface group">
<h2 class="title heading">
      The Complete Works of Emmanuel Allen
    </h2>
<h3 class="byline heading">
<a href="http://archiveofourown.org/users/violue/pseuds/violue" rel="author">violue</a>
</h3>
<div class="summary module" role="complementary">
<h3 class="heading">Summary:</h3>
<blockquote class="userstuff">
<p>Dean Winchester, reluctant business owner, reluctant home owner, and reluctant cat owner, is striking up a very promising friendship with the author of his favorite book series.</p><p>And he has no idea.</p>
</blockquote>
</div>
<div class="notes module" role="complementary">
<h3 class="heading">Notes:</h3>
<blockquote class="userstuff">
<p>Oh yeah, I've got notes.</p><p>
<s>1.) This is complete, though later chapters are still being beta'd. I'll be posting a chapter at a time, whenever the hell I feel like it. Probably every day/every other day because it's hard to just SIT ON ALL THESE CHAPTERS I HAVE WHEN THEY'RE READY TO POST!!!</s>
</p><p>2.) This is of the mostly aimless domestic fluff variety, in that there's no big overarching storyline. But that's pretty common with my stories.  ¯\_(ツ)_/¯ </p><p>3.) There's a bit of <i>me</i> in this story. I am a depressed and surly cat owner living in the Pacific Northwest, and so is Dean, but most of this is just my imagination.</p><p>4.) Thanks to <a href="http://archiveofourown.org/users/Tennyo/works">TENNYO</a>, <a href="http://chiwalker.tumblr.com/">CHIWALKER</a>, <a href="http://buckysbuckhole.tumblr.com/">CASFUCKER</a>, and <a href="http://kelisab.tumblr.com">KELISAB</a> for beta'ing! If you find mistakes in the story, it's all their fault, and you should throw soggy tomatoes at them.</p><p>5.) No, I think that's it. Start reading.</p>
</blockquote>
</div>
</div>
<div class="chapter" id="chapter-1">
<!-- chapter management -->
<div class="chapter preface group" role="complementary">
<h3 class="title">
<a href="/works/6846694/chapters/15628576">Chapter 1</a>: Prologue
    </h3>
<!-- only display byline if different from the main byline -->
</div>
<!--main content-->
<div class="userstuff module" role="article">
<h3 class="landmark heading" id="work">Chapter Text</h3>
<p>"Wow, that’s beautiful!"</p><p>Dean doesn’t even have to look up from his book to know what this customer is talking about. Winchester General Store has a lot of things; food, beer, toiletries, camping gear, used books and more, but the only thing that could be considered "beautiful" in this store is the hand-carved, ornate wooden house sitting in a display case mounted on the wall behind Dean. Actually, "house" isn’t the right word. It started as a house in Dean’s mind, but by the time he was done carving, sanding, polishing, and in some places hot gluing the white oak structure, it had become a mausoleum. A beautiful, <em>inviting </em>mausoleum, but a mausoleum nonetheless. Dean had even borrowed some acrylic paints from Charlie to color the climbing ivy painstakingly carved onto the sides.</p><p>"Thanks, man," Dean says, setting his book down. Might as well let the guy know this was <em>his </em>hard work.</p><p>The man’s eyes widen. "You <em>made </em>this?"</p><p>"Sure did. Worked on it for two months." Dean nods toward the twelve pack of Mountain Dew the customer is holding. "You all set?"</p><p>The man puts the case on the counter by the register, and Dean rings it up. "How much?"</p><p>"Eight ninety-nine for the Dew."</p><p>The man shakes his head. "No, I mean the sculpture. My wife and I just bought a place up in Cougar Falls, and that would look <em>great </em>in the front room."</p><p>Dean blinks, surprised. He’s gotten a lot of compliments on the mausoleum in the past ten or so months, but no one’s ever assumed it was for sale before.</p><p>"Sorry, man, not for sale."</p><p>"Come on. Name your price." Dean gets all sorts of customers here. Locals, people out in the area for camping, people up here to go rafting down Filbert River, and of course, people just passing through on their way to some place bigger and better. This guy falls into the last category.</p><p>"No can do, that thing’s got something important inside. Can’t part with it."</p><p>"Important? Like what?"</p><p>Dean shrugs. "My parents."</p><p>"W… what?" the man stammers.</p><p>"Yeah. There’s an urn inside. Kinda had to glue the top of the building on to get the urn in there, but you can’t really tell unless you’re real close and looking at just the right angle."</p><p>"<em>Both </em>of your parents?"</p><p>"Well, my mom died ages ago, and my dad kept her ashes the rest of his life." Dean turns to look at his carving fondly. "And when my dad died, we had him cremated too. One night I got real drunk, I was still kind of in mourning, and I decided my parents should be together. So I dumped my dad’s ashes into my mom’s urn, and then I gave the urn a good shake," Dean says, shaking an imaginary urn. "My brother was <em>pissed </em>when I told him, but he’s over it now. Anyway, I made this here structure to keep them in. Sort of an apology gift."</p><p>The bell over the front door jingles, and Dean turns back to see the customer has taken off. "Don’t you want your Mountain Dew?" he yells, even though the guy’s already outside.</p><p>Jeez. What a wimp. Dean reaches into the display case, patting the top of the mausoleum gently. "What a baby. Am I right, guys?"</p><p>The urn full of Winchester ashes stays silent of course. Dean snickers, picks his book up off the counter, and gets back to reading.</p><p><br/>
<br/>
</p><p> </p>
</div>
<!--/main-->
</div>

应该是您所需要的.

本文地址:https://www.itbaoku.cn/post/1740071.html