Skip to content

requests: HTTP/1.1 response bodies and max_body limit#1119

Open
pablogventura wants to merge 7 commits into
micropython:masterfrom
pablogventura:requests-http11-essentials
Open

requests: HTTP/1.1 response bodies and max_body limit#1119
pablogventura wants to merge 7 commits into
micropython:masterfrom
pablogventura:requests-http11-essentials

Conversation

@pablogventura
Copy link
Copy Markdown

@pablogventura pablogventura commented Jun 1, 2026

Summary

I use requests on small boards fairly often, and the current package left me hanging on
chunked responses (raw.read() never finished) and still spoke HTTP/1.0 on the wire. This PR
is the first slice of the HTTP work I outlined in #1118: enough client behaviour for real
servers without pulling in TLS or streaming yet.

What I changed:

  • Added requests/_http.py to parse status/headers and read bodies (Content-Length and
    chunked). The full body lands in Response.content so callers do not have to fight the
    socket after the headers.
  • Requests now use HTTP/1.1 with Connection: close (no keep-alive in this PR).
  • resp.headers is a small Headers mapping with case-insensitive keys.
  • Relative Location values on redirects are resolved against the original URL (fixes "requests" does not handle redirection with relative path correctly #931).
  • New max_body argument, default 32 KiB; pass max_body=None if you really want no cap.

Closes #1118.

Testing

I started with the existing socket-mock tests and extended them for bodies, chunked encoding,
redirects, and max_body. On my machine:

python3 -S python-ecosys/requests/test_requests.py

All 14 mock tests pass on CPython 3.12.

I also ran the same file on the unix port (build-standard) and added test_requests_live.py
(local TCP server in a thread) plus run-unix-tests.sh14 + 4 tests, all green.

On hardware I flashed ESP32_GENERIC (ESP32-D0WD, 4 MiB flash), copied this branch’s package
to /lib/requests (the board still ships an older frozen requests, so tests prefer /lib),
and ran run-esp32-tests.sh (mock + live). Same counts, all pass.

Finally I connected the board to my home WiFi and hit real HTTP endpoints (example.com,
httpbin.org GET + redirect) — status 200, bodies as expected. I have not added HTTPS here;
that stays for a follow-up (#838).

CI in this repo should pick up python-ecosys/requests/test_requests.py with the project’s
MicroPython build; the live/ESP32 scripts are mainly for local regression.

Trade-offs and alternatives

Generative AI

I used generative AI tools while working on this PR (including help polishing this
description). I reviewed all of the code and take responsibility for the changes and the
text above.

Introduce read_headers(), read_body(), and chunked decoding
without changing the public request() API yet.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Parse Content-Length and chunked Transfer-Encoding; store body on
Response.content. Default request line to HTTP/1.1. Add max_body
limit (32 KiB default). Fix relative redirect Location URLs.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Cover Content-Length, chunked encoding, max_body, and relative
redirects using the socket mock.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Document HTTP/1.1 response support, max_body, and remaining limits.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Order-independent mock asserts, live TCP tests, unix/ESP32 runners,
prefer /lib over frozen requests on ESP32, and HEAD/auth/encoding fixes.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

requests: HTTP/1.1 response bodies and max_body limit "requests" does not handle redirection with relative path correctly

1 participant