文档地址:http://docs.python-request.org/en/master
Kenneth Reitz: http://www.kennethreitz.org
request库代码:http://github.com/kennethreitz/requests
查看安装哪些库
python freeze
启动terminal。
改自动-启动
这里我选的是cmder,网上有推荐wsl-terminal,为了小简而言,我选择了前者,不过貌似不好看倒是真的
使用参考:https://zhuanlan.zhihu.com/p/28400466
安装地址:http://cmder.net/(电脑上已经有git,所以安装mini版本)
解压后
双击启动exe文件,360会提醒是否阻止,全部不阻止,就可以使用啦
可以使用啦
查看python和pip安装版本,已经安装的包
python --version
pip --version
pip freeze
pip install virtualenv
放在当前目录
virtualenv .env
mac下:
source .env/bin/active
win下:
cd .env\Scripts
activate
pip install requests
可以交互python使用
已经安装过,所以会在cache中,不会耗费网络资源
引入requests库,帮助文档
import requests
help(requests)
“crtl+c”退出
服务端代码:http://httpbin.org
搭载在美国服务器上,访问比较慢
pip install gunicorn httpbin
mac成功
————————————————————————————————————
win下失败截图:
安装包错,百度后说是可能3.5的模块没有该安装包原因所致
先用国内源安装gunicorn
pip install -i https://pypi.mirrors.ustc.edu.cn/simple/ gunicorn
安装到是成功了,但是。。。
启动服务器
gunicorn httpbin:app
在慕课网问答中看到:gunicorn只支持linux系统 不支持windows系统
所以我也很郁闷!
后来找到方法说是网上下载fcntl.py模块放在lib里就可以,但是还有问题……
在github上下载fcntl的安装包:https://github.com/irtnog/dummy-fcntl.py
管理员命令依次执行
python setup.py build
python setup.py install
获取成功
文件夹下增加dist和build目录
——————————————————————————
windows下寻找其他本地搭建服务器环境方法:
方法一:安装Tornado(这个支持python3,安装比较简单)
pip install tornado
方法二:安装twisted(python3.5下报错,无法安装)
百度其他方法安装:
先下载相应安装文件:https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
用pip执行安装whl的命令
A.全局安装方式:
放到C盘根目录下
pip install C:\Twisted-17.9.0-cp35-cp35m-win_amd64.whl
安装成功
B.虚拟环境安装方式:
把whl文件放在虚拟环境下的Script的文件加下
在虚拟环境路径中执行pip安装
pip install Twisted-17.9.0-cp35-cp35m-win_amd64.whl
安装成功
——————————————————————————
mac下:
gunicorn httpbin:app
mac正常访问:http:127.0.0.1:8000
访问成功
win下:
方法一:使用tornado
写一个hello.py的demo
#!/usr/bin/env python
# -*- coding:utf-8 -*-
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/index", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
退回到hello.py所在路径,执行脚本
python hello.py
访问:http://127.0.0.1:8888/index,运行成功
方法二:使用twister
备注:
deactivate
退出虚拟环境
使用curl命令访问网站:
curl --help
curl -v http://www.imooc.com > tmp.txt
response返回内容
启动服务器
退出,执行
##-*- coding:utf-8 -*-
import urllib2
URL_IP = 'HTTP://127.0.0.1:8000/ip'
URL_GET = 'HTTP://127.0.0.1:8000/get'
def use_simple_urllib2():
response = urllib2.urlopen(URL_IP)
print '>>>>Response Headers:'
print response.info()
print '>>>>Response Body:'
print ''.join([line for line in response.readines()])
print '>>>>Response Body:'
print ''.join([line for line in response.readines()])
def use_params_urllib2():
#构建请求参数
params = urllib.urlencode({'param1':'hello','param2':'world'})
print 'Request Params:'
print params
#发送请求
response = urllib2.urlopen('?'.join([URL_GET,'%s']) % params)
#处理响应
print '>>>>Response Headers:'
print response.info()
print '>>>>Status Code:'
print response.getcode()
print '>>>>Response Body:'
print ''.join([line for line in response.readines()])
if __name__ == '__main__':
print ('>>>>User simple urllib2:')
#use_simple_urllib2()
print ('>>>>User params urllib2:')
use_params_urllib2()
python2可使用
#-*- coding:utf-8 -*-
# 以下是python3
import urllib.request
from urllib import request
from urllib import parse
from urllib.request import urlopen
resp = urllib.request.urlopen('http://127.0.0.1:8888/index')
html1 = resp.info()
print('info:')
print(html1)
html2 = resp.getcode()
print('getcode:')
print(html2)
URL_GET = 'http://httpbin.org/get'
params = urllib.parse.urlencode({'param1':'hello','param2':'world'})
response = urllib.request.urlopen('?'.join([URL_GET,'%s']) % params)
print('params:')
print(response.read().decode())
python3可使用,需要先启动tornado
以上为原生url库
reason
状态码
#-*- coding:utf-8 -*-
import requests
URL_IP = 'HTTP://127.0.0.1:8000/ip'
URL_GET = 'HTTP://127.0.0.1:8000/get'
def use_simple_requests():
response = requests.get(URL_IP)
print '>>>>Response Headers:'
print response.headers
print '>>>>Response Body:'
print response.text
def use_params_requests():
#构建请求参数
params = {'param1':'hello','param2':'world'}
print 'Request Params:'
print params
#发送请求
response = requests.get(URL_IP,params = params)
#处理响应
print '>>>>Response Headers:'
print response.headers
print '>>>>Status Code:'
print response.status_code
print response.readon
print '>>>>Response Body:'
print response.json()
python2
#-*- coding:utf-8 -*-
import requests
URL_IP = 'http://httpbin.org/ip'
URL_GET = 'http://httpbin.org/get'
# 以下是python3
def use_simple_requests():
response = requests.get(URL_IP)
print ('>>>>Response Headers:')
print (response.headers)
print ('>>>>Response Body:')
print (response.text)
def use_params_requests():
#构建请求参数
params = {'param1':'hello','param2':'world'}
print ('Request Params:')
print (params)
#发送请求
response = requests.get(URL_IP,params = params)
#处理响应
print ('>>>>Response Headers:')
print (response.headers)
print ('>>>>Status Code:')
print (response.status_code)
print (response.reason)
print ('>>>>Response Body:')
print (response.json)
if __name__ == '__main__':
print ('>>>>User simple requests:')
use_simple_requests()
print ('>>>>User params requests:')
use_params_requests()
python3
Developer:https://developer.github.com/
https://developer.github.com/v3/users
方式1:
方式2:
修改
修改
增加创建
常见异常:
第一个过程等待不超过3秒,第二个等待不超过7秒
两次等待不超过10秒
10秒没通过
没有认证-处理异常
#-*- coding:utf-8 -*-
import json
import requests
from requests import exceptions
URL = 'https://api.github.com'
def build_uri(endpoint):
return '/'.join([URL,endpoint])
def better_print(json_str):
return json.dumps(json.loads(json_str),indent=4)
def request_method():
response = requests.get(build_uri('users/imoocdemo'))
# response = requests.get(build_uri('users/emails'),auth=('imoocdemo','imoocdemo123'))
# print (better_print(response.text))
print (better_print(response.text))
def params_request():
response = requests.get(build_uri('users'),params = {'since':11})
print (better_print(response.text))
print (response.request.headers)
print (response.url)
print(response.status_code)
def json_request():
headers={"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0"}
# headers={"Accept":"application/vnd.github.jean-grey-preview+json"}
# 修改
response = requests.patch(build_uri('user'),auth=('imoocdemo','imoocdemo123'),json={'name':'babymooc2','email':'hello-world@imooc.org'})
# 增加创建
# response = requests.post(build_uri('users/emails'),auth=('imoocdemo','imoocdemo123'),json=['hello'])
print (better_print(response.text))
print (response.request.headers)
print (response.request.body)
print (response.status_code)
print (response.reason)
def timeout_request():
try:
response = requests.get(build_uri('user/emails'),timeout = 10)
response.raise_for_status()
except (exceptions.Timeout) as e:
print (e.message)
except exceptions.HTTPError as e:
#print (e.message)#no
print (e)
else:
print (response.text)
print (response.status_code)
if __name__=='__main__':
# request_method()#可以
# params_request()#可以
# json_request()#422
timeout_request()#404+no attribute
状态码422,请求格式正确,但是由于含有语义错误,无法响应。
经过试验,应当是讲课的人关闭了邮箱认证之类的
使用自己真实可登录的用户名,密码即可修改。查看页面在
https://github.com/你的用户名字
写法问题
避免多次握手
上下问管理
body里面沙溢没有
认证未通过
#-*- coding:utf-8 -*-
URL = 'https://api.github.com'
def build_uri(endpoint):
return '/'.join([URL,endpoint])
def hard_request():
from requests import Request,Session
s = Session()
headers = {'User-Agent':'fake1.3.4'}
req = Request('GET',build_uri('user/email'),auth = ('imoocdemo','imoocdemo123'),headers = headers)
prepped = req.prepare()
print(prepped.body)
print(prepped.headers)
resp = s.send(prepped, timeout=5)
print(resp.status_code)
print(resp.request.headers)
print(resp.text)
if __name__=='__main__':
hard_request()#404和401
404还没查出问题
状态码查找说明(需要访问外国网站):https://zh.wikipedia.org/zh/HTTP%E7%8A%B6%E6%80%81%E7%A0%81
、
utf-8转成unicode格式
#-*- coding:utf-8 -*-
import requests
response = requests.get('https://api.github.com')
response.status_code
response.reason
response.headers
response.url
response.history
response = requests.get('http://api.github.com')
response.history
response = requests.get('https://api.github.com')
dir(response)
response.elapsed
response.request.headers
response.encoding
response.raw.read(10)
response.content
response.json()
response.json()['team_url']
被拒绝:没有权限
查看code和头部信息
开发者工具查看差别,推测可能是user-agent的原因
伪造一个user-agent
状态码改变
试着打印图片
一直在响,乱码,图片不能被打印
支持流传输,写入文件
图片打开了
#-*- coding:utf-8 -*-
import requests
def download_image():
'''
demo:下载图片
:return:
'''
headers = {'User-Agent':'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36'}
url = 'http://pic1.win4000.com/wallpaper/b/5476e87e6d29d.jpg'
response = requests.get(url)
print(response.content)
print(response.status_code,response.reason)
with open('demo.jpg','wb') as fd:
for chunk in response.iter_content(128):
fd.write(chunk)
download_image()
改进的方法:关闭了流,contextlib-上下文语法糖
#-*- coding:utf-8 -*-
import requests
def download_image():
'''
demo:下载图片
:return:
'''
headers = {'User-Agent':'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36'}
url = 'http://pic1.win4000.com/wallpaper/b/5476e87e6d29d.jpg'
response = requests.get(url,headers = headers, stream = True)
print(response.content)
print(response.status_code,response.reason)
with open('demo.jpg','wb') as fd:
for chunk in response.iter_content(128):
fd.write(chunk)
def download_image_improved():
'''
demo:下载图片
:return:
'''
headers = {'User-Agent':'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Mobile Safari/537.36'}
url = 'http://pic1.win4000.com/wallpaper/b/5476e87e6d29d.jpg'
response = requests.get(url)
from contextlib import closing
with closing(requests.get(url,headers = headers, stream = True)) as response:
with open('demo2.jpg', 'wb') as fd:
for chunk in response.iter_content(128):
fd.write(chunk)
# download_image()
download_image_improved()
很多事件基于回调,引起一些列动作
先发,拿到响应后处理
#-*- coding:utf-8 -*-
import requests
def get_key_info(response,*arg,**kwargs):
'''
回调函数
'''
print(response.headers['Content-Type'])
def main():
'''
主程序
:return:
'''
requests.get('https://www.baidu.com',hooks=dict(response = get_key_info))
main()
参考代码:https://github.com/jian-en/imooc-requests
问题本身很复杂比解释的很嘈杂要好
#-*- coding:utf-8 -*-
import requests
BASE_URL = 'https://api.github.com'
def construct_url(end_point):
return '/'.join([BASE_URL, end_point])
def basic_auth():
'''
基本认证
'''
response = requests.get(construct_url('user'),auth=('imoocdemo','imoocdemo123'))
print(response.text)
print(response.request.headers)
basic_auth()
安装bpython
bpython安装参考文档:http://www.cnblogs.com/zhaojiedi1992/p/zhaojiedi_python_006_bpython.html
pip安装成功后,启动报错
原因:缺少一个包
下载curses包地址:https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses
安装成功
修改启动位置
结果运行还是一闪而过,win10,python35的暂时不能用
——————————————————————————————
bpython-交互解释器
base64解码
可以帮助编代码,这个网站相当于对github应用
登录会有个选项,用github方式
获取公共信息
用github举例:
拷贝token
方式一:
获得到信息
#-*- coding:utf-8 -*-
import requests
BASE_URL = 'https://api.github.com'
def construct_url(end_point):
return '/'.join([BASE_URL, end_point])
def basic_auth():
'''
基本认证
'''
response = requests.get(construct_url('user'),auth=('imoocdemo','imoocdemo123'))
print(response.text)
print(response.request.headers)
basic_auth()
方式二:
加了语法糖,优化代码
结果一样
#-*- coding:utf-8 -*-
import requests
BASE_URL = 'https://api.github.com'
def construct_url(end_point):
return '/'.join([BASE_URL, end_point])
# def basic_oauth():
# headers = {'Authorization':'token ed375f28dfcc7d0f410877你自己的token'}
# response = requests.get(construct_url('user/emails'),headers = headers)
# print(response.request.headers)
# print(response.text)
# print(response.status_code)
# basic_oauth()
from requests.auth import AuthBase
class GithubAuth(AuthBase):
def __init__(self,token):
self.token = token
def __call__(self, r):
r.headers['Authorization'] = ' '.join(['token ', self.token])
return r
def oauth_advanced():
auth = GithubAuth('ed375f28dfcc7d0f41087你自己的token')
response = requests.get(construct_url('user/emails'),auth=auth)
print(response.text)
oauth_advanced()
相对基本认证,安全一些,不是明文用户名,密码
安装requeats【socksv5】库的支持
pip install 'requests[socksv5]'
有防火墙
启动代理服务器,成功
#-*- coding:utf-8 -*-
import requests
proxies = {'http':'socks5://127.0.0.1:1080','https':'socks5://127.0.0.1:1080'}
url = 'https://www.facebook.com'
response = requests.get(url, proxies=proxies, timeout=10)
print(response.status_code)
windows上貌似不支持proxies,所以失败
缺点:
A.每次请求都要带着cookie,带宽受影响
B.cookie是在浏览器端的,明文,很多请求可以伪造不安全
存储压力转移到服务器上,安全一些
requests库主要支持是客户端编程
服务端主要是:flask,django等实现cookie和session
用第一原理去思考问题,多问几个为什么,思考底层原理
1.Tornado基本使用:https://www.cnblogs.com/chenchao1990/p/5413547.html