encapsulate accounting calculations in
http_request_stats_bytes_in()
http_request_stats_bytes_out()
more accurate accounting for HTTP/1.1 bytes_in on keep-alive requests
(affects case where client pipelines HTTP/1.1 requests)
remove con->bytes_read and con->bytes_written
(no longer needed since request_st was split from connection struct
and request bytes_read_ckpt and bytes_written_ckpt are maintained
for HTTP/1.x bytes_in and bytes_out accounting. Also, further back,
chunkqueue internal accounting was simplified to maintain bytes_in
and bytes_out to always match chunkqueue length)
call connection_state_machine_h2() only if con->h2 has been initialized
(additional safeguard for #3152)
x-ref:
"Random Segfaults with version 1.4.64 w/ mod_sockproxy and ALPN h2"
https://redmine.lighttpd.net/issues/3152
server.feature-flags += ("auth.delay-invalid-creds" => "enable")
The default is enabled. This feature delays a response to bad http auth
(invalid username or password) to the next second, so up to 1 sec delay.
Delaying the response greatly reduces the efficiency of brute force
password attacks, limiting tries to one-per-second per connection.
Limiting the number of client connections allowed by lighttpd with
server.max-connections sets an upper bound on password tries per second,
but also makes it easier for an attacker to DoS (denial of service) the
server. Therefore, while this mitigation is enabled by default, it can
be disabled with
server.feature-flags += ("auth.delay-invalid-creds" => "disable")
HANDLER_COMEBACK now handled in http_response_handler()
and noinline marked on some routines so that http_response_handler()
has simpler asm for repeated calls by dynamic handlers to generate
content and return HANDLER_WAIT_FOR_EVENT
connection_handle_write_state() call to subrequest handler might set
r->resp_body_finished, but we need to check if r->write_queue is empty
before changing state from CON_STATE_WRITE to CON_STATE_RESPONSE_END
(If r->write_queue was already empty when the subrequest handler marked
resp_body_finished, connection_state_machine_h2() could miss changing
the request state to CON_STATE_RESPONSE_END)
x-ref:
"HTTP/2 requests sometimes take very long (missing last chunk)"
https://redmine.lighttpd.net/issues/3111
"Slow request handling in Lighttpd 1.4.60"
https://redmine.lighttpd.net/boards/2/topics/10063
Existing behavior: mod_proxy *does not* forward Upgrade header
unless explicitly enabled in lighttpd.conf (default: not enabled)
(proxy.header += ("upgrade" => "enable"))
mod_cgi previously used to forward Upgrade request header, but would
remove Upgrade response header if cgi.upgrade was not explicitly enabled
(cgi.upgrade = "enable")
This patch thwarts h2c smuggling when lighttpd.conf has also been
explicitly configured to pass "Upgrade" request header
x-ref:
"h2c Smuggling: Request Smuggling Via HTTP/2 Cleartext (h2c)"
https://labs.bishopfox.com/tech-blog/h2c-smuggling-request-smuggling-via-http/2-cleartext-h2c
reschedule HTTP/2 streams w/ pending data if con->write_queue is empty
reschedule to read deferred (and pending) HTTP/2 frames when
MAX_CONCURRENT_STREAMS are active and a stream completes.
x-ref:
"HTTP/2 requests sometimes take very long (missing last chunk)"
https://redmine.lighttpd.net/issues/3111
allow up to 32k of data frames per stream per round
(previously limited to single max_frame_size (default 16k))
For 8 streams, 32k*8 is 256k, which is current lighttpd MAX_WRITE_LIMIT,
so each stream still gets a chance to write data (unless write queue
not emptied on previous attempt, reducing add limit this round)
cap size of data framed for HTTP/2 response until more data sent to
client
make sure to reschedule connection in job queue if max_bytes reached
and then the entire con->write_queue was flushed to network, or else
there is a chance the request may not get rescheduled (and then will
timeout) if the request is completed from the backend and there is
no other traffic or streams to trigger connection processing.
(check con->write_queue > 8k rather than empty from last round,
since small frames such as connection preface may have been added
this round while processing con->read_queue)
separate internal control for backend max_per_read
When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.
When not streaming, attempt to read and empty kernel socket bufs.
(e.g. MAX_READ_LIMIT 256k)
When writing to sockets (or pipes) attempt to fill kernel socket bufs.
(e.g. MAX_WRITE_LIMIT 256k)
HTTP/2 send GOAWAY soon after client timeout, before potentially
reading new stream requests, which will then have to be reset.
x-ref:
"Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
https://redmine.lighttpd.net/issues/3102
HTTP/2 send GOAWAY soon after keep-alive timeout, before potentially
reading new stream requests, which will then have to be reset.
x-ref:
"Chrome gives random net::ERR_HTTP2_PROTOCOL_ERROR"
https://redmine.lighttpd.net/issues/3102
rename log_error_multiline_buffer() to log_error_multiline()
and take (char *)ptr and (size_t)len instead of (buffer *)b
When debug printing request and response headers,
print each header on separate line for readability
and omit '\r' if "\r\n" ends line
If a system call returns EMFILE, then admin should increase
server.max-fds and check/increase rlimits for num files (ulimit -Hn)
Alternatively, the admin might decrease server.max-connections to limit
the number of connections served in parallel.
merge connection_list_append() into connection_fdwaitqueue_append()
(not converted to singly-linked-list since fdwaitqueue is not expected
to be used during normal operation (except extreme overload condition),
so use allocated list of pointers (allocated when needed) instead of
adding ptr member to (every) allocated struct connection)
remove connections-glue.c
remove inclusion of connections.h by non-base files
srv->lim_conns tracks remaining conns until limit is reached,
replacing (srv->max_conns - srv->conns.used)
srv->srvconf.max_conns is now updated at startup, so
srv->srvconf.max_conns serves as srv->max_conns
keep conns_pool of struct connection separate from conns list
and allocate conns list to srv->srvconf.max_conns size at startup
x-ref:
"Memory fragmentation with HTTP/2 enabled"
https://redmine.lighttpd.net/issues/3084
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.
Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t
* Y2038: lighttpd 1.4.60 and later report Y2038 safety
$ lighttpd -V
+ Y2038 support # Y2038-SAFE
$ lighttpd -V
- Y2038 support (unsafe 32-bit signed time_t) # Y2038-UNSAFE
* Y2038: general platform info
* Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
- all major 64-bit platforms (known to this author) use 64-bit time_t
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
- Linux x32 ABI (different from i686)
- FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
- NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
- OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
- Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
- MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
- e.g. OpenVMS (unknown if lighttpd builds on this platform)
* Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
- Linux 32-bit (including i686)
- glibc 32-bit library support not yet available for 64-bit time_t
- https://sourceware.org/glibc/wiki/Y2038ProofnessDesign
- Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
https://itsubuntu.com/linux-kernel-5-6-to-fix-the-year-2038-issue-unix-y2k/
- https://www.gnu.org/software/libc/manual/html_node/64_002dbit-time-symbol-handling.html
"Note: at this point, 64-bit time support in dual-time
configurations is work-in-progress, so for these
configurations, the public API only makes the 32-bit time
support available. In a later change, the public API will
allow user code to choose the time size for a given
compilation unit."
- compiling with -D_TIME_BITS=64 currently has no effect
- glibc recent (Jul 2021) mailing list discussion
- https://public-inbox.org/bug-gnulib/878s2ozq70.fsf@oldenburg.str.redhat.com/T/
- FreeBSD i386
- DragonFlyBSD 32-bit
* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
* lighttpd prefers system monotonic clock instead of realtime clock
in places where realtime clock is not required
* lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
* (lighttpd presumes that lighttpd will not encounter dates before 1970
during normal operation.)
* lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
to convert negative timestamps for comparisions with 64-bit timestamps
(treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
* lighttpd provides unix_time64_t (int64_t) and
* lighttpd provides struct unix_timespec64 (unix_timespec64_t)
(struct timespec equivalent using unix_time64_t tv_sec member)
* lighttpd provides gmtime64_r() and localtime64_r() wrappers
for platforms 32-bit platforms using 32-bit time_t and
lighttpd temporarily shifts the year in order to use
gmtime_r() and localtime_r() (or gmtime() and localtime())
from standard libraries, before readjusting year and passing
struct tm to formatting functions such as strftime()
* lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
unsigned 32-bit and then to unix_time64_t
* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
on 32-bit platforms using 32-bit signed time_t, underlying libraries and
underlying filesystems might not behave properly after 32-bit signed time_t
overflows (19 Jan 2038 03:14:08 GMT). If a given 32-bit OS does not work
properly using negative time_t values, then lighttpd likely will not work
properly on that system.
* Other references and blogs
- https://en.wikipedia.org/wiki/Year_2038_problem
- https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs
- http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/
This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.
Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.
In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func. In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.
- check for empty strings at config time and set value to NULL if blank
string will be ignored at runtime; at runtime, simple pointer check
for NULL can be used to check for a value that has been set and is not
blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
and use buffer_is_unset() instead of buffer_is_empty(),
where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
truncate string, and use buffer_extend() to extend
Examples where buffer known not to be NULL:
- cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
(though we might set it to NULL if buffer_is_blank(cpv->v.b))
- address of buffer is arg (&foo)
(compiler optimizer detects this in most, but not all, cases)
- buffer is checked for NULL earlier in func
- buffer is accessed in same scope without a NULL check (e.g. b->ptr)
internal behavior change:
callers must not pass a NULL buffer to some funcs.
- buffer_init_buffer() requires non-null args
- buffer_copy_buffer() requires non-null args
- buffer_append_string_buffer() requires non-null args
- buffer_string_space() requires non-null arg
merge connection_accept() into network_server_handle_fdevent()
(possible since connection_accepted() was split out from
connection_accept() a long time ago)
network_server_handle_fdevent() checks max_conns and is the only
callers of connection_accept(), so connection_accept() does not
need to repeat the check.
Note: monotonic time does not change while VM is suspended
Continue to use real time where required by HTTP protocol, for logging
and for other user-visible instances, such as mod_status, as well as for
external databases and caches.
