前言：大概一年前写的，前段时间跑了下，发现还能用，就分享出来了供大家学习，代码的很多细节不太记得了，也尽力做了优化。
因为毕竟是微博，反爬技术手段还是很周全的，怎么绕过反爬的话要在这说都可以单独写几篇文章了（包括网页动态加载，ajax动态请求，token密钥等等，特别是二级评论，藏得很深，记得当时想了很久才成功拿到），直接上代码。
主要实现的功能：
0.理所应当的,绕过了各种反爬。
1.爬取全部的热搜主要内容。
2.爬取每条热搜的相关微博。
3.爬取每条相关微博的评论，评论用户的各种详细信息。
4.实现了自动翻译，理论上来说，是可以拿下与热搜相关的任何细节，但数据量比较大，推荐使用数据库对这个爬虫程序进行优化（因为当时还没学数据库，不会用，就按照一定格式在本地进行了存储）
（未实现功能）：
利用爬取数据构建社交网。可构建python的数据分析，将爬取的用户构成一个社交网络。
项目结构：
weibo.py
用于爬取需要数据，调用回调分析数据后移交给item，再由item移交给管道进行处理，包括持久化数据等等。
import scrapy
from copy import deepcopy
from time import sleep
import json
from lxml import etree
import re
class WeiboSpider(scrapy.Spider):
name = 'weibo'
start_urls = ['https://s.weibo.com/top/summary?Refer=top_hot&topnav=1&wvr=6']
home_page = "https://s.weibo.com/"
#携带cookie发起请求
def start_requests(self):
cookies = "" #获取一个cookie
cookies = {i.split("=")[0]: i.split("=")[1] for i in cookies.split("; ")}
yield scrapy.Request(
self.start_urls[0],
callback=self.parse,
cookies=cookies
)
#分析热搜和链接
def parse(self, response, **kwargs):
page_text = response.text
with open('first.html','w',encoding='utf-8') as fp:
fp.write(page_text)
item = {}
tr = response.xpath('/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# SPIDER_MIDDLEWARES = {
# 'weiboPro.middlewares.WeiboproSpiderMiddleware': 543,
# }
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
'weiboPro.middlewares.WeiboproDownloaderMiddleware': 543,
}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'weiboPro.pipelines.WeiboproPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
scrapy.cfg
配置文件,没啥好写的
[settings]
default = weiboPro.settings
[deploy]
#url = http://localhost:6800/
project = weiboPro
剩下的两个__init__文件空着就行，用不上。
到此这篇关于python实战之Scrapy框架爬虫爬取微博热搜的文章就介绍到这了,更多相关python Scrapy 爬取微博热搜内容请搜索编程网以前的文章或继续浏览下面的相关文章希望大家以后多多支持编程网！
