Python爬虫案例:爬取携程评论

前言

之前爬取美团,马蜂窝等网站的数据都挺顺利,大众点评(这个反爬机制有点麻烦)在磕磕绊绊中也算成功(重点是网页页数的变化和关键字的隐藏替换)但携程居然遇到了瓶颈。

主要是查看源代码时发现关键商户信息根本就找不到,就很奇怪。对于关键信息评论发现翻页时网页的url不变,网上查了一下说是使用是动态的网址进行建构的,Ajax页面加载,那么通用的request.get()就不能用了,所以采取模拟浏览器进行数据爬取。

为什么选取龟峰呢?因为正好才从龟峰回来,这几乎是去过的人最少的5A级景区,大胆猜测数据可能没有那么多(相比去的其他地方而言)而且去的时候惊喜的给免了门票,所以用这个来测试。

PS:如有需要Python学习资料的小伙伴可以加下方的群去找免费管理员领取

 

可以免费领取源码、项目实战视频、PDF文件等

 

第一步:打开网页

 

可能需要登录后才能查看评价

 

查看网页源代码

 

可以看到无法加载出关键评论信息

F12查看评论的具体代码,如下图找到评论列表,找到url和request payload

 

 

至此已经获取了网址最重要的内容

直接上完整代码

import requests import json import time import csv import re c=open(r'D:\guifeng.csv','a+',newline='',encoding='utf-8') fieldnames=['user','time','score','content'] writer=csv.DictWriter(c,fieldnames=fieldnames) writer.writeheader() head = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0'} postUrl = "https://sec-m.ctrip.com/restapi/soa2/12530/json/viewCommentList" data_1 = { "pageid": "290510", "viewid": "127481", "tagid": "-11", "pagenum": "1", "pagesize": "10", "contentType": "json", "SortType": "1", "head": { "appid": "100013776", "cid": "09031010311083118856", "ctok": "", "cver": "1.0", "lang": "01", "sid": "8888", "syscode": "09", "auth": "", "extension": [ { "name": "protocal", "value": "https" } ] }, "ver": "7.10.3.0319180000" } html = requests.post(postUrl, data=json.dumps(data_1)).text html = json.loads(html) jingqu = '龟峰' pages = html['data']['totalpage'] datas = [] for j in range(pages): data1 = { "pageid": "290510", "viewid": "127481", "tagid": "-11", "pagenum": str(j + 1), "pagesize": "10", "contentType": "json", "SortType": "1", "head": { "appid": "100013776", "cid": "09031010311083118856", "ctok": "", "cver": "1.0", "lang": "01", "sid": "8888", "syscode": "09", "auth": "", "extension": [ { "name": "protocal", "value": "https" } ] }, "ver": "7.10.3.0319180000" } datas.append(data1) for k in datas[:50]: print('正在抓取第' + k['pagenum'] + "页") time.sleep(3) html1 = requests.post(postUrl, data=json.dumps(k)).text html1 = json.loads(html1) comments = html1['data']['comments'] for i in comments: user = i['uid'] time1 = i['date'] score = i['score'] content = i['content'] content = re.sub(" ", "", content) content = re.sub(" ", "", content) writer.writerow({'user': user,'time':time1,'score': score,'content':content}) c.close()

 

按F5直接运行得出文件(自动存储在D盘里)

 

右键以记事本打开(如果使用excel需要先另存为否则初次打开会是乱码)

 

关键步骤解读

postUrl = "https://sec-m.ctrip.com/restapi/soa2/12530/json/viewCommentList"

这个网址是评论的通用网址不需要改动

可以改动的地方为:

c=open(r'D:\guifeng.csv','a+',newline='',encoding='utf-8')

存储的路径可以自定义,这里我直接用的是D盘

data_1 = {
    "pageid": "290510", "viewid": "127481", "tagid": "-11", "pagenum": "1", "pagesize": "10", "contentType": "json", "SortType": "1", "head": { "appid": "100013776", "cid": "09031010311083118856", "ctok": "", "cver": "1.0", "lang": "01", "sid": "8888", "syscode": "09", "auth": "", "extension": [ { "name": "protocal", "value": "https" } ] }, "ver": "7.10.3.0319180000" }

具体数据就是如下图需爬取网页评论代码里红圈的部分

 

同样的这个也是

  data1 = {
        "pageid": "290510", "viewid": "127481", "tagid": "-11", "pagenum": str(j + 1), "pagesize": "10", "contentType": "json", "SortType": "1", "head": { "appid": "100013776", "cid": "09031010311083118856", "ctok": "", "cver": "1.0", "lang": "01", "sid": "8888", "syscode": "09", "auth": "", "extension": [ { "name": "protocal", "value": "https" } ] }, "ver": "7.10.3.0319180000" } datas.append(data1)

注意:

 "pagenum": str(j + 1),

这一步不可更改

最后

for k in datas[:50]: print('正在抓取第' + k['pagenum'] + "页") time.sleep(3)

这里的爬取的页数50是可以更改的

时间间隔3也是可以更改的

至此可以成功爬取携程上任意一家的评论,yes!!!

本文的文字及图片来源于网络,仅供学习、交流使用,不具有任何商业用途,版权归原作者所有,如有问题请及时联系我们以作处理。

作者:一千口水牛