Commit Graph

4088 Commits

Author SHA1 Message Date
Glenn Strauss 624d66b625 [core] add wolfssl-specific include
2021-09-08 15:06:06 -04:00
Glenn Strauss 7386502572 [build] update ax_prog_cc_for_build.m4

  "Cross Compile with ptxdist failed for 1.4.59 (1.4.58 works)"
2021-09-08 15:06:06 -04:00
Glenn Strauss 89c97b8cd8 [core] conns_pool separate from conns list (#3084)
keep conns_pool of struct connection separate from conns list
and allocate conns list to srv->srvconf.max_conns size at startup

  "Memory fragmentation with HTTP/2 enabled"
2021-09-08 15:06:06 -04:00
Glenn Strauss a8c9b38118 [core] adjust srv->srvconf.max_conns at startup
adjust srv->srvconf.max_conns at startup, if adjustment needed
2021-09-08 15:06:06 -04:00
Glenn Strauss c69450ec04 [core] free fdwaitqueue list when empty
fdwaitqueue is typically unused except in severe overload conditions,
so free up the list storage as soon as it is empty
2021-09-08 15:06:06 -04:00
Glenn Strauss 311d4b0ef4 [core] adjust server overload check
do not re-enable sockets if there are any connections in fdwaitqueue
(server_overload_check() now better mirrors server_load_check())
2021-09-08 15:06:06 -04:00
Glenn Strauss 600dfba3e6 [core] set min srv->max_fds = 32 (sanity check)
(server load checks will fail if value is too low)
2021-09-08 15:06:06 -04:00
Glenn Strauss 6515f77665 [core] clarify error message in gw_backend.c
clarify error message in gw_backend.c if connect() to unix socket fails
2021-09-08 15:06:06 -04:00
Glenn Strauss 8c36615f85 [mod_cgi] cgi.limits "read-timeout" "write-timeout" (#3086)
implement write(), read() timeouts for CGI


  "sockets disabled, out-of-fds with proxy module"
2021-09-08 15:06:06 -04:00
Glenn Strauss da562e3fd6 [core] http_response_read() indicate resp finished
return HANDLER_FINISHED from http_response_read() if response finished,
whether due to reading EOF (prior behavior), or if Content-Length was
provided and we have finished reading Content-Length, or if a module
sets r->resp_body_finished for any other reason.  This may save an
unnecessary poll() and read() to receive EOF when Content-Length has
already been read.
2021-09-08 15:06:06 -04:00
Glenn Strauss 39a577985f [core] improve handling of suboptimal backend wr
more efficiently handle reading of suboptimal backend write behavior

check to accumulate small reads in mem before flushing to temp file
2021-09-08 15:06:06 -04:00
Glenn Strauss 833d658729 [core] http_response_append_{buffer,mem}()
manage r->resp_body_scratchpad in new funcs
rather than
which now only decode chunked encoding, more apropos for the func names
2021-09-08 15:06:06 -04:00
Glenn Strauss 2ef31a1b3f [core] chunkqueue_append_buffer always clears buf
chunkqueue_append_buffer() always clears buffer
(instead of relying on caller to do so after the call)
2021-09-08 15:06:06 -04:00
Glenn Strauss 3f17f94336 [core] tweak buffer merging to reduce mem
tweak buffer merging in chunkqueue_append_buffer() to trade off
a slight increase in copying for slightly more constrained mem usage
2021-09-08 15:06:06 -04:00
Glenn Strauss d958cf3262 [doc] 2021-09-08 15:06:06 -04:00
Glenn Strauss 02646ea2ad [core] connect, write, read timeouts on backends (fixes #3086)
implement connect(), write(), read() timeouts on backends


  "sockets disabled, out-of-fds with proxy module"
2021-09-08 15:06:06 -04:00
Glenn Strauss 4f96dac841 [core] gw_backend_error() shared code 2021-09-04 08:08:27 -04:00
Glenn Strauss ae149c42e5 [core] store time for last r/w to a backend socket 2021-09-04 08:08:26 -04:00
Glenn Strauss 075230158a [core] make missing mod_deflate not a fatal error
mod_compress was removed in lighttpd 1.4.56, subsumed by mod_deflate.

distros may package mod_deflate separately from the lighttpd package.

However, existing configurations may reference mod_compress.
lighttpd maps the reference from mod_compress to mod_deflate,
but after a system is upgraded to lighttpd 1.4.56 or later,
mod_compress may have been removed, and mod_deflate -- which
might be a separate package -- might not be installed.

lighttpd will still issue error trace about the missing mod_deflate
modules, as well as about the unrecognized configuration directives
(compress.* or deflate.*), but this will no longer be a fatal error.
2021-09-04 08:08:26 -04:00
Glenn Strauss 39d399112a [mod_openssl] no ALPN fatal error w/ mod_sockproxy (fixes #3081)
If mod_sockproxy -- or other connection-level handler -- has been set
on the request prior to mod_openssl processing TLS Client Hello, then
failure to match ALPN protocol is no longer treated as a TLS connection
setup error.

  "sockproxy: Do not validate ALPN protocols"
2021-09-04 08:08:26 -04:00
Glenn Strauss 9a2404cec6 [core] quiet compiler warnings
cast away signedness warning in request_check_hostname()
mod_gnutls https_add_ssl_client_entries crts allocated if crt_size != 0
  (which is already checked earlier in routine)

report from FaceBook Infer static analysis tool (
- quiet dead store warnings
- check return != NULL from allocation funcs
2021-09-04 08:08:26 -04:00
Glenn Strauss 8492d0947a [build] check for sys/filio.h in CMake and meson
present in and SConstruct,
but missing from src/CMakeLists.txt and src/
2021-09-04 08:08:26 -04:00
Glenn Strauss 71931d9238 [core] define _DEFAULT_SOURCE in first.h 2021-09-04 08:08:26 -04:00
Glenn Strauss 92576a2f4b [core] Y2038: use _TIME_BITS=64 on 32-bit glibc
Y2038: use _TIME_BITS=64 on 32-bit glibc platforms

2021-09-04 08:08:26 -04:00
Glenn Strauss 1ccc1d4ff5 [core] log_clock_gettime->clock_gettime for 64-bit
map log_clock_gettime() directly to clock_gettime() for 64-bit time_t
(on systems with clock_gettime())
2021-09-04 08:08:26 -04:00
Glenn Strauss 87b09d144e [core] use CLOCK_MONOTONIC_COARSE where available
server.c uses monotonic clock to detect change in second,
so CLOCK_MONOTONIC_COARSE clock resolution (often ~1ms)
is more than sufficient.  (Obtaining CLOCK_MONOTONIC_COARSE
can be faster than CLOCK_MONOTONIC.)

2021-09-04 08:08:26 -04:00
Glenn Strauss dbf7588147 [core] tune http_response_process_headers()
- rearrange some code for better CPU cache use
- use http_header_str_contains_token()
2021-09-04 08:08:26 -04:00
Glenn Strauss f4ff56e064 [mod_deflate] use http_header_str_contains_token 2021-09-04 08:08:26 -04:00
Glenn Strauss 309c1693ac [multiple] Y2038 32-bit signed time_t mitigations
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.

Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t

* Y2038: lighttpd 1.4.60 and later report Y2038 safety
  $ lighttpd -V
    + Y2038 support                                    # Y2038-SAFE
  $ lighttpd -V
    - Y2038 support (unsafe 32-bit signed time_t)      # Y2038-UNSAFE

* Y2038: general platform info
  * Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
      - all major 64-bit platforms (known to this author) use 64-bit time_t
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
      - Linux x32 ABI (different from i686)
      - FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
      - NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
      - OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
      - Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
        Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
      - MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
      - e.g. OpenVMS (unknown if lighttpd builds on this platform)
  * Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
      - Linux 32-bit (including i686)
          - glibc 32-bit library support not yet available for 64-bit time_t
              - Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
                "Note: at this point, 64-bit time support in dual-time
                 configurations is work-in-progress, so for these
                 configurations, the public API only makes the 32-bit time
                 support available. In a later change, the public API will
                 allow user code to choose the time size for a given
                 compilation unit."
              - compiling with -D_TIME_BITS=64 currently has no effect
          - glibc recent (Jul 2021) mailing list discussion
      - FreeBSD i386
      - DragonFlyBSD 32-bit

* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
  * lighttpd prefers system monotonic clock instead of realtime clock
    in places where realtime clock is not required
  * lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
  * (lighttpd presumes that lighttpd will not encounter dates before 1970
    during normal operation.)
  * lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
    to convert negative timestamps for comparisions with 64-bit timestamps
    (treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
  * lighttpd provides unix_time64_t (int64_t) and
  * lighttpd provides struct unix_timespec64 (unix_timespec64_t)
    (struct timespec equivalent using unix_time64_t tv_sec member)
  * lighttpd provides gmtime64_r() and localtime64_r() wrappers
    for platforms 32-bit platforms using 32-bit time_t and
    lighttpd temporarily shifts the year in order to use
    gmtime_r() and localtime_r() (or gmtime() and localtime())
    from standard libraries, before readjusting year and passing
    struct tm to formatting functions such as strftime()
  * lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
    unsigned 32-bit and then to unix_time64_t

* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
  on 32-bit platforms using 32-bit signed time_t, underlying libraries and
  underlying filesystems might not behave properly after 32-bit signed time_t
  overflows (19 Jan 2038 03:14:08 GMT).  If a given 32-bit OS does not work
  properly using negative time_t values, then lighttpd likely will not work
  properly on that system.

* Other references and blogs
2021-09-04 08:08:26 -04:00
Glenn Strauss 82c199db3f [autoconf] add AC_SYS_LARGEFILE for lfs
add AC_SYS_LARGEFILE for large file support
(in addition to manually defining macros for large file support
 on different platforms; bootstrap-and-suspenders)
2021-09-04 08:08:26 -04:00
Glenn Strauss 3ce554e18c [core] report Y2038 support in lighttpd -V
(report unsafe 32-bit signed time_t)

64-bit time_t (default with 64-bit builds) is safe beyond Y2038
2021-09-04 08:08:26 -04:00
Glenn Strauss 9c9cd37320 [tests] speed up mod-fastcgi and mod-scgi tests
On platforms where lighttpd gets SIGCHLD,
lighttpd can quickly detect exit of backend process
and can restart backend process
2021-09-04 08:08:26 -04:00
Glenn Strauss 5294d7e18f [tests] combine *.t using tests/condition.conf
combine tests/*.t using tests/condition.conf into tests/core-condition.t

Platforms which are horrifically slow starting processes (e.g. Windows)
take much more time to start and stop lighttpd many times for
independent *.t instances run through the Perl Test::More framework
2021-09-04 08:08:26 -04:00
Glenn Strauss f472611597 [tests] combine *.t using tests/lighttpd.conf
combine tests/*.t using tests/lighttpd.conf into tests/request.t

Platforms which are horrifically slow starting processes (e.g. Windows)
take much more time to start and stop lighttpd many times for
independent *.t instances run through the Perl Test::More framework
2021-09-04 08:08:26 -04:00
Glenn Strauss 91472ab768 [tests] t/test_mod_staticfile
move some tests from tests/request.t to src/t/test_mod_staticfile.c
2021-09-04 08:08:26 -04:00
Glenn Strauss a0a8cf821d [tests] mv tests from request.t to test_request.c
move some tests from tests/*.t to src/t/test_request.c
2021-08-27 02:16:54 -04:00
Glenn Strauss d4f43f7e1b [tests] update t/test_request.c
use http_header_request_get() in tests to retrieve request headers
2021-08-27 02:16:54 -04:00
Glenn Strauss f7bebe99a5 [tests] use generated date in HTTP If conditionals
replace hard-coded date in HTTP If conditional tests

  "Make tests pass after 2035"

github: closes #106
2021-08-27 02:16:54 -04:00
Glenn Strauss 361b96476e [core] replace strncasecmp w/ buffer_eq_icase_ssn
replace strncasecmp() w/ buffer_clen() and buffer_eq_icase_ssn()
(portability; remove use of alt sys-strings.h portability header)
2021-08-27 02:16:54 -04:00
Glenn Strauss 75e9b2857a [core] better trace if TLS received on clear port 2021-08-27 02:16:54 -04:00
Glenn Strauss 6881f79c32 [mod_auth] detect and skip BWS (bad whitespace)
detect and skip BWS (bad whitespace) in Authorization
2021-08-27 02:16:54 -04:00
Glenn Strauss 7a913fb153 [mod_deflate] improve compress.*->deflate.* remap 2021-08-27 02:16:54 -04:00
Glenn Strauss a3603074c1 [mod_deflate] use brotli quality 5 by default
BROTLI_DEFAULT_QUALITY is 11, which may lead to a higher compression
ratio, but potentially at a cost of taking *many* multiples of the
time taken to compress at quality level 5.

2021-08-27 02:16:54 -04:00
Glenn Strauss 3f248f0900 [mod_deflate] deflate.params per-encoder params
mechanism to define per-encoder params
parsed into structured data at startup

compression level is the initial target
deflate.params is a better solution to the deflate.compression-level,
which is a single range 1-9 that is overload and applied to all encoders
without any scaling, even though encoders might have different scales.

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 777d659c5c [mod_deflate] use ZSTD_c_strategy w/ compress lvl
map deflate.compression-level to ZSTD_c_strategy as the current
enum for ZSTD_strategy is in the range 1 .. 9, which matches the
gzip compression level 1 .. 9 range

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 7759660de5 [mod_deflate] use deflate.allowed-encodings order
use deflate.allowed-encodings ordering as server preference list
when choosing an encoding from list in client request Accept header

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 09aa794cae [core] refactor config_check_cond_nocache() flow 2021-08-27 02:16:54 -04:00
Glenn Strauss a3f5fa3ff6 [core] config_check_cond_nocache() xor return code 2021-08-27 02:16:54 -04:00
Glenn Strauss 9fe8fbaa72 [multiple] http_method_buf()
- http_method_buf() returns (const buffer *)
- comment out unused get_http_status_name()
- inline func for http_append_method()

config processing requires a persistent buffer for method on the
off-chance that the config performed a capturing regex match in
$HTTP["method"] condition and used it later (e.g. in mod_rewrite)
(Prior behavior using r->tmp_buf was undefined in this case)
2021-08-27 02:16:54 -04:00
Glenn Strauss 1de03a03ee [core] reduce $HTTP["host"] compare str scanning 2021-08-27 02:16:54 -04:00