Python爬虫

> 参考慕课网的爬虫教程

# 简单的爬虫架构
![title](https://leanote.com/api/file/getImage?fileId=583d337bab644170320002b3)

* 爬虫调度端：启动、停止、监听爬虫
* 爬虫程序：
    - url管理器：管理已爬取ur和待爬取的url，取出一个待爬取url送入网页下载器
    - 网页下载器：将url指定的网页下载存储成字符串，将字符串送入网页解析器
    - 网页解析器：从字符串中解析出有价值的信息和数据，并爬取取出其中其他网页的url，送入url管理器
    - 三个模块形成循环
* 得到价值数据

![title](https://leanote.com/api/file/getImage?fileId=583d3559ab6441378101b914)

# URL管理器
![title](https://leanote.com/api/file/getImage?fileId=583d3608ab644170320002fd)

# 网页下载器
![title](https://leanote.com/api/file/getImage?fileId=583d3732ab6441378101b952)

* 第一种方法
```python
import urllib2
# 直接请求
response = urllib2.urlopen('http://www.baidu.com')
# 获取状态码
print response.getcode()
# 读取内容
cont = response.read()
```
* 第二种方法
```python
import urllib2
# 创建Request对象
request = urllib2.Request(url)
# 添加数据
request = request.add_data('a','1')
# 添加http的head
request = add_header('User-Agent','Mozilla/5.0')
# 发送请求获取结果
response = urllib2.urlopen(request)
```
* 第三种方法
![title](https://leanote.com/api/file/getImage?fileId=583d3b72ab6441378101b9c2)
```python
import urllib2,cookielib
# 创建cookie容器
cj = cookielib.CookieJar()
# 创建一个opener
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
# 给urllib2安装opener
urllib2.install_opener(opener)
# 使用带有cookie的urllib2访问网页
response = urllib2.urlopen("http://www.baidu.com")
```
# 网页解析器
![title](https://leanote.com/api/file/getImage?fileId=583d648dab644170320009cd)
![title](https://leanote.com/api/file/getImage?fileId=583d6508ab644170320009d0)
![title](https://leanote.com/api/file/getImage?fileId=583d667dab6441378101c06d)

* `pip install beautifulsoup4` 安装beautifulsoup4
* 创建BeautifulSoup对象
```python
from bs4 import BeautifulSoup
soup = BeautifulSoup(
 html_doc, # HTML文档字符串
 'html.parser' # HTML解析器
 from_encoding='utf8' # HTML文档的编码
)
```
* 搜索节点(find_all, find)
```python
# find_all(name, attrs, string) 
# args: 节点标签名称, 节点属性, 节点内容
# 查找所有标签为a的节点
soup.find_all('a')
# 查找所有标签为a,链接符合/view/123.htm形式的节点
soup.find_all('a', href='/view/123.htm')
soup.find_all('a', href=re.compile(r'/view/\d+\.htm'))
# 查找所有标签为div, class为abc, 文字为Python的节点
# 因为python中有class的关键字，所以bs中class属性以class_表示
soup.find_all('a', class_='abc', string='Python')
```
* 访问节点信息
```python
# 得到节点：<a href='1.html'>Python</a>
# 获取查找到的节点的标签名称
node.name
# 获取查找到的节点的href属性
node['href']
# 获取查找到的节点的节点内容
node.get_text()
```
* 示例
```python
# coding:utf-8
import re
from bs4 import BeautifulSoup
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
The Dormouse's story

Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.

...
"""
soup = BeautifulSoup(html_doc, 'html.parser', from_encoding='utf-8')

print '获取所有连接'
links = soup.find_all('a')
for link in links:
    print link.name, link['href'], link.get_text()
    
print '获取lacie的连接'
link_node = soup.find('a', href="http://example.com/lacie")
print link_node.name, link_node['href'], link_node.get_text()

print '正则匹配'
link_node = soup.find('a', href=re.compile(r"ill"))
print link_node.name, link_node['href'], link_node.get_text()

print '获取P段落文字'
p_node = soup.find('p', class_='title')
print p_node.name, p_node['class'], p_node.get_text()
```

# 实例爬虫
![title](https://leanote.com/api/file/getImage?fileId=583d750fab6441378101c1ee)
![title](https://leanote.com/api/file/getImage?fileId=583d7625ab64417032000b89)

逆风起笔