Commit Graph

4110 Commits (614250b4d82b7c471cc68ba011e7b3d658417f22)

Author SHA1 Message Date
Glenn Strauss 309c1693ac [multiple] Y2038 32-bit signed time_t mitigations
Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.

Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t

* Y2038: lighttpd 1.4.60 and later report Y2038 safety
  $ lighttpd -V
    + Y2038 support                                    # Y2038-SAFE
  $ lighttpd -V
    - Y2038 support (unsafe 32-bit signed time_t)      # Y2038-UNSAFE

* Y2038: general platform info
  * Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
      - all major 64-bit platforms (known to this author) use 64-bit time_t
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
      - Linux x32 ABI (different from i686)
      - FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
      - NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
      - OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
      - Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
        Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
      - MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
  * Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
      - e.g. OpenVMS (unknown if lighttpd builds on this platform)
  * Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
      - Linux 32-bit (including i686)
          - glibc 32-bit library support not yet available for 64-bit time_t
              - Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
                "Note: at this point, 64-bit time support in dual-time
                 configurations is work-in-progress, so for these
                 configurations, the public API only makes the 32-bit time
                 support available. In a later change, the public API will
                 allow user code to choose the time size for a given
                 compilation unit."
              - compiling with -D_TIME_BITS=64 currently has no effect
          - glibc recent (Jul 2021) mailing list discussion
      - FreeBSD i386
      - DragonFlyBSD 32-bit

* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
  * lighttpd prefers system monotonic clock instead of realtime clock
    in places where realtime clock is not required
  * lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
  * (lighttpd presumes that lighttpd will not encounter dates before 1970
    during normal operation.)
  * lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
    to convert negative timestamps for comparisions with 64-bit timestamps
    (treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
  * lighttpd provides unix_time64_t (int64_t) and
  * lighttpd provides struct unix_timespec64 (unix_timespec64_t)
    (struct timespec equivalent using unix_time64_t tv_sec member)
  * lighttpd provides gmtime64_r() and localtime64_r() wrappers
    for platforms 32-bit platforms using 32-bit time_t and
    lighttpd temporarily shifts the year in order to use
    gmtime_r() and localtime_r() (or gmtime() and localtime())
    from standard libraries, before readjusting year and passing
    struct tm to formatting functions such as strftime()
  * lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
    unsigned 32-bit and then to unix_time64_t

* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
  on 32-bit platforms using 32-bit signed time_t, underlying libraries and
  underlying filesystems might not behave properly after 32-bit signed time_t
  overflows (19 Jan 2038 03:14:08 GMT).  If a given 32-bit OS does not work
  properly using negative time_t values, then lighttpd likely will not work
  properly on that system.

* Other references and blogs
2021-09-04 08:08:26 -04:00
Glenn Strauss 82c199db3f [autoconf] add AC_SYS_LARGEFILE for lfs
add AC_SYS_LARGEFILE for large file support
(in addition to manually defining macros for large file support
 on different platforms; bootstrap-and-suspenders)
2021-09-04 08:08:26 -04:00
Glenn Strauss 3ce554e18c [core] report Y2038 support in lighttpd -V
(report unsafe 32-bit signed time_t)

64-bit time_t (default with 64-bit builds) is safe beyond Y2038
2021-09-04 08:08:26 -04:00
Glenn Strauss 9c9cd37320 [tests] speed up mod-fastcgi and mod-scgi tests
On platforms where lighttpd gets SIGCHLD,
lighttpd can quickly detect exit of backend process
and can restart backend process
2021-09-04 08:08:26 -04:00
Glenn Strauss 5294d7e18f [tests] combine *.t using tests/condition.conf
combine tests/*.t using tests/condition.conf into tests/core-condition.t

Platforms which are horrifically slow starting processes (e.g. Windows)
take much more time to start and stop lighttpd many times for
independent *.t instances run through the Perl Test::More framework
2021-09-04 08:08:26 -04:00
Glenn Strauss f472611597 [tests] combine *.t using tests/lighttpd.conf
combine tests/*.t using tests/lighttpd.conf into tests/request.t

Platforms which are horrifically slow starting processes (e.g. Windows)
take much more time to start and stop lighttpd many times for
independent *.t instances run through the Perl Test::More framework
2021-09-04 08:08:26 -04:00
Glenn Strauss 91472ab768 [tests] t/test_mod_staticfile
move some tests from tests/request.t to src/t/test_mod_staticfile.c
2021-09-04 08:08:26 -04:00
Glenn Strauss a0a8cf821d [tests] mv tests from request.t to test_request.c
move some tests from tests/*.t to src/t/test_request.c
2021-08-27 02:16:54 -04:00
Glenn Strauss d4f43f7e1b [tests] update t/test_request.c
use http_header_request_get() in tests to retrieve request headers
2021-08-27 02:16:54 -04:00
Glenn Strauss f7bebe99a5 [tests] use generated date in HTTP If conditionals
replace hard-coded date in HTTP If conditional tests

  "Make tests pass after 2035"

github: closes #106
2021-08-27 02:16:54 -04:00
Glenn Strauss 361b96476e [core] replace strncasecmp w/ buffer_eq_icase_ssn
replace strncasecmp() w/ buffer_clen() and buffer_eq_icase_ssn()
(portability; remove use of alt sys-strings.h portability header)
2021-08-27 02:16:54 -04:00
Glenn Strauss 75e9b2857a [core] better trace if TLS received on clear port 2021-08-27 02:16:54 -04:00
Glenn Strauss 6881f79c32 [mod_auth] detect and skip BWS (bad whitespace)
detect and skip BWS (bad whitespace) in Authorization
2021-08-27 02:16:54 -04:00
Glenn Strauss 7a913fb153 [mod_deflate] improve compress.*->deflate.* remap 2021-08-27 02:16:54 -04:00
Glenn Strauss a3603074c1 [mod_deflate] use brotli quality 5 by default
BROTLI_DEFAULT_QUALITY is 11, which may lead to a higher compression
ratio, but potentially at a cost of taking *many* multiples of the
time taken to compress at quality level 5.

2021-08-27 02:16:54 -04:00
Glenn Strauss 3f248f0900 [mod_deflate] deflate.params per-encoder params
mechanism to define per-encoder params
parsed into structured data at startup

compression level is the initial target
deflate.params is a better solution to the deflate.compression-level,
which is a single range 1-9 that is overload and applied to all encoders
without any scaling, even though encoders might have different scales.

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 777d659c5c [mod_deflate] use ZSTD_c_strategy w/ compress lvl
map deflate.compression-level to ZSTD_c_strategy as the current
enum for ZSTD_strategy is in the range 1 .. 9, which matches the
gzip compression level 1 .. 9 range

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 7759660de5 [mod_deflate] use deflate.allowed-encodings order
use deflate.allowed-encodings ordering as server preference list
when choosing an encoding from list in client request Accept header

  "ModDeflate questions (possibly some feature requests too)"
2021-08-27 02:16:54 -04:00
Glenn Strauss 09aa794cae [core] refactor config_check_cond_nocache() flow 2021-08-27 02:16:54 -04:00
Glenn Strauss a3f5fa3ff6 [core] config_check_cond_nocache() xor return code 2021-08-27 02:16:54 -04:00
Glenn Strauss 9fe8fbaa72 [multiple] http_method_buf()
- http_method_buf() returns (const buffer *)
- comment out unused get_http_status_name()
- inline func for http_append_method()

config processing requires a persistent buffer for method on the
off-chance that the config performed a capturing regex match in
$HTTP["method"] condition and used it later (e.g. in mod_rewrite)
(Prior behavior using r->tmp_buf was undefined in this case)
2021-08-27 02:16:54 -04:00
Glenn Strauss 1de03a03ee [core] reduce $HTTP["host"] compare str scanning 2021-08-27 02:16:54 -04:00
Glenn Strauss 4b9da9f1e8 [core] parse $HTTP["remote-ip"] CIDR mask at start
parse $HTTP["remote-ip"] CIDR mask into structured data at startup

note: adds buffer_move() to configparser.y to reduce memory copying
for all config values, and is required for remote-ip to preserve the
structured data added after the config value string.  (Alternatively,
could have normalized the remote-ip value after copying into dc->string)
2021-08-27 02:16:54 -04:00
Glenn Strauss e34ce5f217 [core] const buffer * in config_check_cond_nocache
use (const buffer *) in config_check_cond_nocache()
2021-08-27 02:16:54 -04:00
Glenn Strauss 1ca721d479 [core] reduce excess cc inlining in http_chunk.c 2021-08-27 02:16:54 -04:00
Glenn Strauss 3a568c639c [core] http_date_time_append() convenience macro
append IMF-fixdate time string to (buffer *)
2021-08-27 02:16:54 -04:00
Glenn Strauss 9f90f33288 [core] write_all() simpler loop; better asm 2021-08-27 02:16:54 -04:00
Glenn Strauss 18e96334d6 [core] proxy_create_env() tweaks
reorder some code for better asm

proxy_set_Forwarded() sets multiple request headers, and does so prior
to walking all request headers to create request to backend.  This is
done so that specific already-existing request headers from client are
overwritten (intentionally) in proxy_set_Forwarded().

Expect header is handled, but not expected since client-sent Expect
header is handled (and unset) in connection_handle_read_post_state()
2021-08-27 02:16:54 -04:00
Glenn Strauss 9f82ba8fab [mod_proxy] proxy_response_headers load v earlier
proxy_response_headers() issue variable load slightly earlier
2021-08-27 02:16:54 -04:00
Glenn Strauss f2cdee2430 [core] order gw_host members for packing and usage 2021-08-27 02:16:54 -04:00
Glenn Strauss 50f88480a6 [core] order gw_proc members for packing and usage 2021-08-27 02:16:54 -04:00
Glenn Strauss e89d75136c [core] avoid inlining buffer_eq_icase_ssn()
(slightly better asm)
2021-08-27 02:16:54 -04:00
Glenn Strauss 2899d7f983 [core] construct file path after docroot hook
also remove some excess/duplicative trace during request processing
2021-08-27 02:16:54 -04:00
Glenn Strauss 5bc92071d7 [core] reset large path buffers from long URLs 2021-08-27 02:16:54 -04:00
Glenn Strauss f1e8a82f1a [multiple] inline struct in con->dst_addr_buf
(mod_extforward recently changed to use buffer_move() to save addr
 instead of swapping pointers)
2021-08-27 02:16:54 -04:00
Glenn Strauss e6f286ef5f [multiple] reduce use of BUFFER_INTLEN_PTR
reduce use of BUFFER_INTLEN_PTR where b->ptr known not to be NULL
2021-08-27 02:16:54 -04:00
Glenn Strauss 38a9f0ebdd [mod_access] remove excess trace
trace is still issued if access is denied
  and r->log_request_handling is set
2021-08-27 02:16:53 -04:00
Glenn Strauss 2d9bf8062d [mod_extforward] inline some more data structures 2021-08-27 02:16:53 -04:00
Glenn Strauss 4ccd86f981 [core] allocate initial request pool w/ srv->conns
allocate reusable h2 request pool when initial srv->conns pool allocated
2021-08-27 02:16:53 -04:00
Glenn Strauss 1cd73b08a6 [core] move backtrace and assert macros to ck.[ch] 2021-08-27 02:16:53 -04:00
Glenn Strauss 4e5b822b16 [mod_magnet] cache script objects at config time
always check at least mtime and size to trigger script reload
2021-08-27 02:16:53 -04:00
Glenn Strauss f490078d0f [multiple] buffer_copy_string_len_lc()
convenience wrapper combining
and making a single pass over string
2021-08-27 02:16:53 -04:00
Glenn Strauss 8c5acaae3c [core] lowercase r->http_host, r->uri.authority
always lowercase r->http_host, r->uri.authority for consistency
2021-08-27 02:16:53 -04:00
Glenn Strauss 84fa4b8d1b [core] buffer_realloc() using power-2 realloc 2021-08-27 02:16:53 -04:00
Glenn Strauss caff799ccb [build] look for port.h on Solaris, not sys/port.h 2021-08-27 02:16:53 -04:00
Glenn Strauss 0f90a9e320 [core] reduce memcmp in http_request_parse_header
extend http_header_parse_ctx to support enum http_header_h2_e
HTTP/2 pseudo-headers in hpctx->id before using memcmp()
2021-08-27 02:16:53 -04:00
Glenn Strauss 5beee8b2d4 [core] buffer_path_simplify() quick(er) path
scan to detect (potential) need for path simplification
(repeated '/' or "/.") before copying each char in string
2021-08-27 02:16:53 -04:00
Glenn Strauss 1f161bda1f [build] look for memcpy and define HAVE_MEMCPY
(HAVE_MEMCPY checked by ancient MD5 implementation in algo_md5.c)
2021-08-27 02:16:53 -04:00
Glenn Strauss 56e2881c36 [mod_auth] refactor mod_auth_check_basic()
refactor mod_auth_check_basic()
- use stack for base64-decoded username:password, and limit to 1k
2021-08-27 02:16:53 -04:00
Glenn Strauss 20c56cecc3 [mod_auth] refactor mod_auth_check_digest()
refactor mod_auth_check_digest()
- smaller functions
- collect parsed Authorization header into http_auth_digest_params_t
- use string references rather than copying and modifying Authorization
2021-08-27 02:16:53 -04:00