Python Requests library: Make HTTP requests elegant

Requests is one of the most popular HTTP libraries in the Python ecosystem. Its design philosophy is "HTTP for Humans" - simple and intuitive, allowing you to focus on business logic rather than tedious network details. Compared to the built-inurllib, the interface of Requests is more user-friendly and can easily handle various HTTP requests and responses.

Whether you are developing crawlers, calling RESTful APIs, or need to interact with third-party services, Requests can greatly improve your development efficiency.

1. Installation and Startup

Use pip to complete the installation:

pip install requests

If you are using the Anaconda environment, Requests usually comes pre-installed. If you encounter permission issues, you can add--userParameters or run pip in a virtual environment:

pip install --user requests

Tip: Recommended in a virtual environment (such asvenv、conda) to avoid polluting system-level Python.

Once installed, the import library is ready to use:

import requests

2. Send the first GET request

The most common operation in HTTP is the GET request, which is used to obtain resources. It only takes one line of code to make a request:

response = requests.get('https://www.example.com/')

After the request is completed, what we are most concerned about is usually the status code and response content:

print(response.status_code)   # 200 表示成功
print(response.text)          # 响应的 HTML 或文本内容

.status_code: HTTP status code,200represents success,404means not found,500It's a server error etc.
.text: The automatically decoded text content of Requests is usually a readable string.

2.1 Carrying query parameters

Many APIs require passing query string parameters. Requests can receive a dictionary and automatically concatenate it into URL parameters:

params = {'q': 'python', 'page': 1}
response = requests.get('https://www.example.com/search', params=params)

print(response.url)  # 实际请求地址：https://www.example.com/search?q=python&page=1

This way I don’t have to manually splice?q=python&page=1, which is both clear and avoids encoding problems.

3. Processing of response content

Requests provides multiple ways to extract response data, suitable for different scenarios:

# 原始字节数据（适用于图片、文件等）
content = response.content

# 自动解码后的文本
text = response.text

# 若响应是 JSON，直接解析成字典
data = response.json()

# 查看当前使用的字符编码
print(response.encoding)  # 通常为 'utf-8'

# 必要时可以手动指定编码
response.encoding = 'utf-8'

.content:returnbytes, suitable for processing binary files such as images and PDFs.
.text: Returns a string, Requests will be based on the response header orencodingProperties are automatically decoded.
.json(): If the response body is in JSON format, you can directly call this method to get the Python data structure, eliminating the need to manuallyjson.loads()。

4. Custom request header

Many scenarios require disguising User-Agent or adding authentication tokens, etc. passheadersParameters can be easily set:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)',
    'Authorization': 'Bearer your_token_here'
}
response = requests.get('https://api.example.com/data', headers=headers)

The case of keys in Headers does not matter, Requests are automatically processed into standard format.

5. Send POST request

POST requests are used to submit data, such as login, form submission, or resource creation.

5.1 Form data

submitapplication/x-www-form-urlencodedFormatted data (emulates HTML form):

data = {'username': 'admin', 'password': 'secret'}
response = requests.post('https://example.com/login', data=data)

5.2 Submit JSON data

Nowadays, most REST APIs use JSON and can be used directlyjsonParameters, Requests will be set automaticallyContent-Type: application/json：

json_data = {'name': 'Alice', 'age': 30}
response = requests.post('https://api.example.com/users', json=json_data)

5.3 File upload

When uploading files, usefilesparameters, and open the file in binary mode:

with open('report.pdf', 'rb') as f:
    files = {'file': f}
    response = requests.post('https://example.com/upload', files=files)

Notice'rb'mode, otherwise the file may be corrupted due to encoding issues.filesThe dictionary supports uploading multiple files at the same time.

6. Other HTTP methods

Requests provides corresponding shortcut functions for common HTTP methods:

# 更新资源 (PUT)
response = requests.put('https://api.example.com/item/1', data={'name': 'new'})

# 删除资源 (DELETE)
response = requests.delete('https://api.example.com/item/1')

# 获取响应头信息 (HEAD)
response = requests.head('https://www.example.com')

# 查询服务器支持的请求方法 (OPTIONS)
response = requests.options('https://api.example.com')

These methods are used withget()andpost()Highly consistent and common parameters.

7. Response headers and Cookies

7.1 View response headers

Response headers are stored in dictionary form inresponse.headers, keys are not case-sensitive:

print(response.headers['Content-Type'])  # 'text/html; charset=utf-8'
print(response.headers.get('Server'))    # 获取服务器类型

7.2 Handling Cookies

Cookies returned by the server can be passedresponse.cookiesAccess; custom cookies can also be carried when sending requests:

# 获取响应中的 Cookie
print(response.cookies['session_id'])

# 在请求中带上 Cookie
cookies = {'session_id': 'abc123'}
response = requests.get('https://example.com/dashboard', cookies=cookies)

8. Timeout control

Network requests may remain blocked due to network fluctuations or server unresponsiveness. set uptimeoutYou can limit the waiting time to avoid program freezes:

try:
    response = requests.get('https://example.com', timeout=3)  # 3 秒超时
except requests.exceptions.Timeout:
    print("请求超时，请检查网络或增加超时时间")

It is recommended to explicitly set the timeout in all requests to improve the robustness of the program.

9. Maintain session: Session object

If you need to request the same website multiple times and keep logged in status or common request headers, you can useSessionObject:

session = requests.Session()
session.headers.update({'User-Agent': 'MyApp/1.0'})

# 登录（假设 POST 后会设置 session cookie）
session.post('https://example.com/login', data={'user': 'me', 'pass': '123'})

# 后续请求自动携带 cookie 和 headers
response = session.get('https://example.com/profile')

SessionThe bottom layer uses a connection pool and reuses TCP connections across requests to improve performance.

10. Quick overview of advanced functions

10.1 Redirect processing

By default, Requests automatically follow 3xx redirects. If you want to disable automatic jump, you can setallow_redirects=False：

response = requests.get('https://example.com', allow_redirects=False)
# 查看重定向历史
print(response.history)

10.2 Using a proxy

passproxiesParameter configuration HTTP/HTTPS proxy:

proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080',
}
response = requests.get('https://api.example.com', proxies=proxies)

10.3 SSL Certificate Verification

Requests verifies SSL certificates by default when accessing HTTPS sites. If you encounter a self-signed certificate in a development environment, you can temporarily turn off verification (Do not do this in a production environment), or specify a custom CA certificate:

# 临时关闭验证
response = requests.get('https://example.com', verify=False)

# 使用自定义 CA 证书
response = requests.get('https://example.com', verify='/path/to/cert.pem')

11. Best practices for error handling

Abnormalities are inevitable in network operations. Perfect error handling can make the program more reliable:

try:
    response = requests.get('https://example.com', timeout=5)
    response.raise_for_status()   # 检查 HTTP 状态码，4xx/5xx 会抛出异常
except requests.exceptions.HTTPError as e:
    print(f"HTTP 错误: {e}")
except requests.exceptions.ConnectionError as e:
    print(f"连接错误: {e}")
except requests.exceptions.Timeout as e:
    print(f"请求超时: {e}")
except requests.exceptions.RequestException as e:
    print(f"其他请求异常: {e}")

raise_for_status()Will be thrown when status code is 4xx or 5xxHTTPError, eliminating manual inspection.
All Requests exceptions inherit fromrequests.exceptions.RequestException。

12. Performance optimization tips

12.1 Streaming download of large files

When downloading large files, usestream=TrueYou can write to disk while downloading to avoid loading into memory at once:

response = requests.get('https://example.com/bigfile.zip', stream=True)
with open('bigfile.zip', 'wb') as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

12.2 Adjust connection pool

by customizingHTTPAdapter, you can control the connection pool size and retry strategy:

from requests.adapters import HTTPAdapter

session = requests.Session()
adapter = HTTPAdapter(
    pool_connections=10,
    pool_maxsize=10,
    max_retries=3
)
session.mount('https://', adapter)
session.mount('http://', adapter)

This is very useful for high concurrent request scenarios.

13. Summary

Requests encapsulates the complexity of HTTP behind a simple API, allowing you to:

Use very few codes to complete common operations such as GET and POST;
Easily handle JSON, file uploads, cookies and headers;
Maintain login status and reuse connections through Session;
Use mechanisms such as timeout, exception-handling, and streaming requests to build stable network programs.

After mastering these core functions, you can use Requests comfortably in actual projects. If you want to explore further, it is recommended to consult the official documentation and the "HTTP Definitive Guide" to understand the underlying protocol - of course, Requests is elegant and powerful enough for daily development.

#Python Requests library: Make HTTP requests elegant

#1. Installation and Startup

#2. Send the first GET request

#2.1 Carrying query parameters

#3. Processing of response content

#4. Custom request header

#5. Send POST request

#5.1 Form data

#5.2 Submit JSON data

#5.3 File upload

#6. Other HTTP methods

#7. Response headers and Cookies

#7.1 View response headers

#7.2 Handling Cookies

#8. Timeout control

#9. Maintain session: Session object

#10. Quick overview of advanced functions

#10.1 Redirect processing

#10.2 Using a proxy

#10.3 SSL Certificate Verification

#11. Best practices for error handling

#12. Performance optimization tips

#12.1 Streaming download of large files

#12.2 Adjust connection pool

#13. Summary