request

size\r\n data\r\n data\r\n … 0\r\n\r\n(end)

ConnectionError: ('Connection aborted.', BadStatusLine("''",))

req = request.Request('GET', 'http://localhost:8000/?u=8') pre = pre.prepare() pre.url = u'http://localhost:8000/?u=8 8' request.session().send(pre)

Traceback (most recent call last): File "<pyshell#30>", line 1, in <module> s.send(pr) File "C:\Python27\lib\site-packages\requests\sessions.py", line 573, in send r = adapter.send(request, **kwargs) File "C:\Python27\lib\site-packages\requests\adapters.py", line 415, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', BadStatusLine("''",))

a adpters has a PoolManager and maybe one or two ProxyManager(each for http, https)

s = requests.session() r = s.get('http://httpbin.org/relative-redirect/2')

resp1 = r.history¹ resp2 = r.history² resp3 = r

req1 = resp1.request req2 = resp2.request req3 = resp3.request

[r for r in s.resolve_redirects(resp1, req1)]

create a Session s mount adapters https:// http:// to HTTPAdapter()

Session send a request s.request create a Request req send the request s.send adapter send the request, return the Response r

• adapter *

** CA certificate bundle try: from certifi import where except ImportError: def where(): """Return the preferred certificate bundle."""

return os.path.join(os.path.dirname(file), 'cacert.pem')

attrs = [ 'headers', 'cookies', 'auth', 'proxies', 'hooks', 'params', 'verify', 'cert', 'prefetch', 'adapters', 'stream', 'trust_env', 'max_redirects', ]

METHOD:

request : prepare_request merge_{environmentsetting} send *send : get_adapter resolve_redirects *resolve_redirects(generator) : rebuild_proxies rebuild_auth send (redirect=False) get_adapter

get post delete head put patch options

merge_{environmentsettings} mount prepare_request rebuild_auth rebuild_proxies close

COOKIES:

1. self.cookies = cookiejar_fromdict({})
2. cookies (a param)

prepare_request merge_cookies self.cookies, cookies (param), RequestsCookieJar()

send extract_cookiestojar(self.cookies, request, resp.raw)

resolve_redirects extract_cookiestojar(prepare_request._cookies, req, resp.raw prepared_request._cookies.update(self.cookies) prepared_request.prepare_cookies(prepared_request._cookies)

ps : send also invocate Respose.content

content (in bytes) text (in unicode) json

send() conn = sellf.get_connection() resp = conn.urlopen() low_conn = conn._getconn low_conn.send() build_response()

build_response()

METHOD init_poolmanager

send : get_connection cert_verify request_url add_headers build_response

get_connection : proxy_managerfor

add_headers build_response cert_verify close

proxy_headers proxy_managerfor request_url

PoolManager，proxyManager中的pools类似字典，以(scheme, host, port)为key， 根据scheme的不同，对应值为HTTPConnection和HTTPSConnection.

主要的方法有: connection_fromhost(self, host, port=None, scheme='http') PoolManager直接根据(scheme,host,port)从pools取出ConnectPool，如 果没有就创建一个新的 proxyManager重写了该方法， 对scheme=https直接使用(scheme, host, port), 对scheme=http使用(porxy.scheme, proxy.host, porxy.port)

_new_pool(self, scheme, host, port)

urlopen(self, method, url, redirect=True, **kw) 调用connect_{fromhost获得ConnectionPool，再调用ConnectionPool} 和它同名的方法urlopen 并对重定向做了一些处理(requests没有使用)

HTTP(S)ConnectionPool 维持一个由HTTP(S)Connection组成的pool，一个thread-safe的队列，其中 每个Conneciton完全一样

HTTPConnectionPool维持代理信息， 但HTTPConnectPool不对代理做任何处 理 HTTPSConnectionPool在生成新的Connection时，以代理的主机名和端口作为 参数 ConnectionCls(proxy.host, proxy.port)

urlopen调用httplib.Connection.request 和 getresponse 如何有代理，调用self._prepareproxy()

http = urllib3.PoolManager( cert_reqs='CERT_REQUIRED', # Force certificate check. ca_certs=certifi.where(), # Path to the Certifi bundle. )

ca_carts = DEFAULT_CABUNDLEPATH = certs.where() # requests/cert.py

r = requests.get('http://www.baidu.com') print r.text

Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'gbk' codec can't encode character u'\xbb' in position 23594

illegal multibyte sequence

requests/adapters.py

def proxy_headers(self, proxy): """Returns a dictionary of the headers to add to any request sent through a proxy. This works with urllib3 magic to ensure that they are correctly sent to the proxy, rather than in a tunnelled request if CONNECT is being used.

This should not be called from user code, and is only exposed for use when subclassing the

:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.

:param proxies: The url of the proxy being used for this request.

:param kwargs: Optional additional keyword arguments. """

private key is created like this: openssl genrsa -out ryans-key.pem 2048

Certificates are public keys signed by a Certificate Authority orself-signed. The first step to getting a certificate is to create a "Certificate Signing Request" (CSR) file. openssl req -new -sha256 -key ryans-key.pem -out ryans-csr.pem

To create a self-signed certificate with the CSR openssl x509 -req -in ryans-csr.pem -signkey ryans-key.pem -out ryans-cert.pem

Alternatively you can send the CSR to a Certificate Authority for signing.

参见nodejs的 api document(TLS,HTTPS) 和 python的Library Reference(ssl)

node tls.createServer(options[, secureConnectionListener]) tls.connect(options[, callback]) options.key 私钥 options.cert 证书 options.ca 受信证书 options.rejectUnauthorized 是否拒绝任何未被认证的连接

python ssl.wrap_socket(sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE(certificates ignored), CERT_REQUIRED (required and validated). ca_certs=None, …)

requests 的大部分功能都通过一个API解决 Session.request

这些参数一部分是为了构建请求request： 请求的方法method，url，query params，头headers，cookies；请求体 data，也可以是json格式，上传文件 files，简单的HTTP认证auth. 另一部分参数有超时控制timeout，代理proxies，SSL认证verify，cert，钩 子函数hooks，是否允许重定向allow_{redirects，还有一个参数stream不知} 道干球用的

而这个函数是Session的一个实例函数，Session就像他的名字指示的一样， 模拟一个会话过程。一个session对象有着和 request 函数参数差不多的实 例变量，由此来控制会话期间每个请求的默认行为。

通过用户提供给request函数的参数先构造一个Request对象，在经过一些处 理（参数的检查,合并一些session属性…），生成preparedRequest对象， 选取合适 的适配器（http or https）发送请求，返回Response.

直接将proxies参数传给adapter，由adapter使用urllib3.ProxyManager完 成后续工作

adapter.send(request, proxies) conn = adapter.get_connection(request.url, proxies) url = adapter.request_url(request, proxies) resp = conn.urlopen(request.method, url)

两种情况 s = requests.session() s.proxies = {'http': '127.0.0.1:8000', 'https': '127.0.0.1:8000'} s.get('http://www.google.com/t') s.get('https://www.google.com/ts')

代理 import socket sock = socket.socket() sock.bind(('127.0.0.1',8000)) sock.listen(5) while 1: r,_ = sock.accept() print r.recv(65536) r.sendall('HTTP/1.0 200 OK\r\n\r\nHello World\r\n') r.close()

18832691616

Question: httplib.HTTPConnection 支持CONNECT tunnel urllib3.HTTPConnection urllib3.HTTPConnectionPool

>>> proxy = urllib3.ProxyManager('http://localhost:3128/') >>> r1 = proxy.request('GET', 'http://google.com/') >>> r2 = proxy.request('GET', 'http://httpbin.org/') >>> len(proxy.pools) 1 >>> r3 = proxy.request('GET', 'https://httpbin.org/') >>> r4 = proxy.request('GET', 'https://twitter.com/') >>> len(proxy.pools) 3

目的是http的，返回一个 HTTPConnectPool(proxy.host, proxy.port, connection_poolkw) 并且发送请求时使用full URL

目的是https的，返回一个 HTTPSConnectPool(dest.host, dest.port, connection_poolkw) connect_{poolkw包含代理信息} VerifiedHTTPSConnect会调用_{tunnel使用代理} 注意：httplib.HTTPSConnection中 self._tunnelhost 指的是目的服务器，而 self.host指的是代理服务器

def connection_fromhost(self, host, port=None, scheme='http'): if scheme == "https": return super(ProxyManager, self).connection_fromhost( host, port, scheme)

return super(ProxyManager, self).connection_fromhost( self.proxy.host, self.proxy.port, self.proxy.scheme)

与HTTPS相关的参数有verify，cert

cert是客户端证书，不常用

verify的默认值是True，表示对服务器的证书进行认证，受信证书(CA)使用默认 值 DEFAULT_CABUNDLEPATH(requests/cacert.pem)

verify的值为False，表示不对服务器证书进行认证，等于把ssl.wrap_socket的 参数cert_{reqs设为ssl}.CERT_NONE

verify的值还可以是路径，表示对服务器的证书进行验证，并且受信证书(CA)使 用该路径指定的证书文件

VerifiedHTTPSConnection 对httplib.HTTPSConnection.connect方法进行了重写，使用ssl……

类变量定义在所有方法之外，被所有实例对象所共享。但是一旦实例对象直 接对类变量修改或赋值，或者说实例对象创建了一个与类变量同名的实例变 量，这样就把类变量给覆盖了

class Dog:

kind = 'canine' # class variable shared by all instances

def __init__{(self, name)}: self.name = name # instance variable unique to each instance

>>> d = Dog('Fido') >>> e = Dog('Buddy')

>>> print d.kind, e.kind canine canine

>>> e.kind = 'E' >>> Dog.kind = "wowo"

>>> print d.kind, e.kind wowo E

在类中定义的函数是 unbound method，实例化对象后，这个方法成为这个对象 的bound method. 方法调用时，从类中搜索.

>>> class T: def f(self): print 'Hello World'

>>> t = T()

>>> T.f <unbound method T.f>

>>> t.f <bound method T.f of <__main__.T instance at 0x02780E18>>

>>> t.f() Hello World

>>> def f(self): print 'FUCK'

>>> T.f = f

>>> t.f() FUCK

将一个函数变为bound method，只需要调用函数对象的的__get__方法

>>> t.g = f.__get__(t) >>> t.g() FUCK