Commit Graph

4119 Commits (0a5cbc62bdf21e550b152be389c83631d726afa1)

Author SHA1 Message Date
Glenn Strauss 0a5cbc62bd [mod_magnet] rename magnet_cgi_ to magnet_envvar_
rename magnet_cgi_* to magnet_envvar_* to be less confusing
The (array *)r->env is populated with connection-level env (e.g TLS)
but is not populated with CGI/1.1 environment variables.
Values similar to most CGI/1.1 environment variables are already available
from lighty.r.req_attr[] (or lighty.env[]) which are prepared on demand.
2021-09-08 15:06:07 -04:00
Glenn Strauss c0ea7f8e27 [mod_magnet] use blank str for nil (do not panic)
substitute blank string for nil param (instead of lua script panic)
2021-09-08 15:06:07 -04:00
Glenn Strauss 3ddf457560 [mod_magnet] lighty.c.readdir
lighty.c.readdir           dir walk

Note: the "lighty.c.*" namespace is EXPERIMENTAL / UNSTABLE
In the future, these may be removed, altered, or moved to a different
2021-09-08 15:06:07 -04:00
Glenn Strauss ed94ae88e8 [mod_magnet] lighty.c.cookie_tokens
lighty.c.cookie_tokens     parse cookie header into table

Note: the "lighty.c.*" namespace is EXPERIMENTAL / UNSTABLE
In the future, these may be removed, altered, or moved to a different
2021-09-08 15:06:07 -04:00
Glenn Strauss c82ca970fb [core] refuse excess h2 streams at con start (fixes #3093)
refuse excess h2 streams up to stream id 199 within 1-2 seconds of
establishing connection.  This attempts to more gracefully handle
excessively agressive browsers sending too many new requests *before*
processing the initial SETTINGS frame sent by the server.

lighttpd HTTP/2 support limits SETTINGS_MAX_CONCURRENT_STREAMS to 8
and communicates this in the initial SETTINGS sent by the server.

  "Chrome 92, HTTP/2, fcgi, mutiple puts no response"
  "POST request DATA part for non-existing URI closes HTTP/2 connection prematurely"
2021-09-08 15:06:07 -04:00
Glenn Strauss aa1027ba24 [mod_magnet] lighty.r.* interfaces to request
lighty.r.req_header[]     HTTP request headers
lighty.r.req_attr[]       HTTP request attributes / components
lighty.r.req_env[]        HTTP request environment variables
lighty.r.resp_header[]    HTTP response headers
lighty.r.resp_body.*      HTTP response body accessors
lighty.r.resp_body.len    HTTP response body length
lighty.r.resp_body.add()  HTTP response body add (string or table)
lighty.r.resp_body.set()  HTTP response body set (string or table)

lighty.r.req_header[] allows get/set of request headers
If modifications would affect config processing, script should return
lighty.MAGNET_RESTART to have lighttpd restart the modified request
This differs from the older lighty.env[] table, which (previously) did
not permit modification of request headers.

lighty.r.resp_header[] allows get/set of response headers
(certain connection-level headers such as Connection and
Transfer-Encoding are restricted from modification)
This differs from the older lighty.header[] table, which is
collected and deferred, being applied after the script exits.

lighry.r.resp_body.* adds/sets response body content.
This differs from the older lighty.content[] table, which is
collected and deferred, being applied after the script exits.

  r = lighty.r
  resp_header = r.resp_header
  resp_header["Content-Type"] = "text/html"
  lighty.r.resp_header["Content-Type"] = "text/html"
older syntax (less clear)
  lighty.header["Content-Type"] = "text/html"
2021-09-08 15:06:07 -04:00
Glenn Strauss 86c87d3c6d [core] use mod name in trace instead of mod_gw 2021-09-08 15:06:07 -04:00
Glenn Strauss a3b76ed5c4 [core] reuse some cold duplicate hdr match code 2021-09-08 15:06:07 -04:00
Glenn Strauss b596680ba4 [mod_magnet] look up env id by strlen, then strcmp 2021-09-08 15:06:07 -04:00
Glenn Strauss 614250b4d8 [mod_magnet] expose enc/dec str funcs to lua
lighty.c.b64urldec         validate and decode base64url
lighty.c.b64urlenc         base64url encode, no padding
lighty.c.b64dec            validate and decode base64
lighty.c.b64enc            base64 encode, no padding
lighty.c.hexdec            validate and decode hex str
lighty.c.hexenc            uc; lc w/ lua s = s:lower()
lighty.c.xmlenc            xml-encode/html-encode: <>&'\"`
lighty.c.urldec            url-decode
lighty.c.urlenc            url-encode
lighty.c.urldec_query      url-decode query-string
lighty.c.urlenc_query      url-encode query-string
lighty.c.urlenc_normalize  url-encode normalization
lighty.c.fspath_simplify   simplify fspath

Note: the "lighty.c.*" namespace is EXPERIMENTAL / UNSTABLE
In the future, these may be removed, altered, or moved to a different
2021-09-08 15:06:07 -04:00
Glenn Strauss 76ed1f4462 [mod_magnet] protect and control lighty table mod
protect lighty table against mistaken modification
(lighty table is resused between script invocations)

detect if lighty.content[] is set by script (or if not set)
2021-09-08 15:06:07 -04:00
Glenn Strauss 188248d238 [mod_magnet] lighty.stat now returns userdata obj
lighty.stat now returns a userdata object instead of a populated table.
The userdata object provides methods to access the (stat_cache_entry *)
stored in the userdata object.  (This approach is often much faster than
populating the table of stat entries, as the fields get copied on demand
(upon use) into lua types.)
2021-09-08 15:06:07 -04:00
Glenn Strauss bf05943258 [mod_magnet] allow modification of request headers
allow modification of request headers, with some limitations:
- lighttpd config conditions are not reset;
  lua script must return lighty.RESTART_REQUEST to reprocess request
  (if applicable to the running lighttpd config)
- lighttpd config request header policy is not applied;
  lua script must not set unvalidated, untrusted, or non-normalized vals
- Host may not be unset
- Content-Length may not be modified
- some hop-by-hop (connection) level headers may not be modified
  (e.g. Connection, Transfer-Encoding)
2021-09-08 15:06:07 -04:00
Glenn Strauss b5cdc958a7 [mod_magnet] expose md and hmac funcs to lua"algo", "data")
lighty.c.hmac("algo", "secret", "data")

"algo" can be one of: "md5", "sha1", "sha256", "sha512"
(as long as lighttpd compiled w/ crypto lib supporting those algorithms)

lighty.c.digest_eq("digest1", "digest2")
- performs a timing-safe, case-insensitive comparison of two hex digests
- "digest1" and "digest2" are hex strings (of binary digests)
- returns boolean true or false

lighty.c.secret_eq("data1", "data2")
- performs a timing-safe comparison of two strings
  (and attempts to hides differences in string lengths)
- "data1" and "data2" are strings
- returns boolean true or false

- cached time(); seconds since 1 Jan 1970 00:00:00 GMT
  (faster than os.time())

- generate pseudo-random number

Note: the "lighty.c.*" namespace is EXPERIMENTAL / UNSTABLE
In the future, these may be removed, altered, or moved to a different
2021-09-08 15:06:07 -04:00
Glenn Strauss d97eeefa34 [core] li_hmac_sha512() 2021-09-08 15:06:07 -04:00
Glenn Strauss 347479573a [mod_magnet] reuse lighty lua table
reuse lighty lua table; leave on stack
reset response tables lighty.header[] and lighty.content[] each request
2021-09-08 15:06:07 -04:00
Glenn Strauss 77ea7d8a56 [mod_extforward] HAProxy PROXY env PP2_UNIQUE_ID
propagate PP2_TYPE_UNIQUE_ID into request env as PP2_UNIQUE_ID
2021-09-08 15:06:07 -04:00
Glenn Strauss 24d103c5c5 [mod_indexfile] section into subroutines
separate out logic to choose whether or not to try to handle request
from logic to try to find target from list of potential indexfiles
2021-09-08 15:06:07 -04:00
Glenn Strauss 67c0b1498a [multiple] remove base.h include where not used
(substitute request.h if file only accesses request_st,
 and not connection or server structs)
2021-09-08 15:06:07 -04:00
Glenn Strauss a04d69eaaa [mod_uploadprogress] use splay_tree for req list
(avoids persistent memory allocation for list struct)
(reduce possibility of long-term memory fragmentation due to
2021-09-08 15:06:06 -04:00
Glenn Strauss 7e000de04e [mod_cgi] use linked list for process list
(avoids persistent memory allocation for list struct)
(reduce possibility of long-term memory fragmentation due to mod_cgi)
2021-09-08 15:06:06 -04:00
Glenn Strauss 7b3a4f13d7 [mod_cgi] reuse chunk buffers
reuse chunk buffers rather than persistent allocated memory
(reduce possibility of long-term memory fragmentation due to mod_cgi)
2021-09-08 15:06:06 -04:00
Glenn Strauss 55d8ed1b25 [core] remove HANDLER_WAIT_FOR_FD
If a system call returns EMFILE, then admin should increase
server.max-fds and check/increase rlimits for num files (ulimit -Hn)

Alternatively, the admin might decrease server.max-connections to limit
the number of connections served in parallel.
2021-09-08 15:06:06 -04:00
Glenn Strauss 50a274312d [core] reduce optim inline of cold funcs 2021-09-08 15:06:06 -04:00
Glenn Strauss 38ecede057 [core] clarify an error message 2021-09-08 15:06:06 -04:00
Juliusz Sosinowicz db54269c06 [mod_wolfssl] wolfSSL_sk_X509_NAME_push change
adjust code for wolfSSL_sk_X509_NAME_push() interface change in
wolfssl v4.8.0

[gstrauss: edited var names, whitespace, comments, commit message]

  "[wolfssl] Correct return check for wolfSSL_sk_X509_NAME_push"

github: closes #107
2021-09-08 15:06:06 -04:00
Glenn Strauss 62a9d5b78f [core] clear request,connection pools every 64 sec (#3084)
  "Memory fragmentation with HTTP/2 enabled"
2021-09-08 15:06:06 -04:00
Glenn Strauss 323e03fb2d [core] remove connection_list_append()
merge connection_list_append() into connection_fdwaitqueue_append()

(not converted to singly-linked-list since fdwaitqueue is not expected
to be used during normal operation (except extreme overload condition),
so use allocated list of pointers (allocated when needed) instead of
adding ptr member to (every) allocated struct connection)

remove connections-glue.c
remove inclusion of connections.h by non-base files
2021-09-08 15:06:06 -04:00
Glenn Strauss dc2d1dfe47 [core] change con joblist to singly-linked-list
avoids separate memory allocation for list of pointers

adds ability to check if con is already in joblist,
so do not re-add con if already in joblist

since con is checked if in joblist before being added to joblist,
there is no longer need for two lists and jobs can be processed
before poll() for to process new events
2021-09-08 15:06:06 -04:00
Glenn Strauss 81a107b4e6 [core] change srv->conns to doubly-linked-list
avoids separate memory allocation for list of pointers
2021-09-08 15:06:06 -04:00
Glenn Strauss 5a58f6963c [core] rename srv->max_conns -> srv->lim_conns
srv->lim_conns tracks remaining conns until limit is reached,
replacing (srv->max_conns - srv->conns.used)

srv->srvconf.max_conns is now updated at startup, so
srv->srvconf.max_conns serves as srv->max_conns
2021-09-08 15:06:06 -04:00
Glenn Strauss 624d66b625 [core] add wolfssl-specific include
2021-09-08 15:06:06 -04:00
Glenn Strauss 7386502572 [build] update ax_prog_cc_for_build.m4

  "Cross Compile with ptxdist failed for 1.4.59 (1.4.58 works)"
2021-09-08 15:06:06 -04:00
Glenn Strauss 89c97b8cd8 [core] conns_pool separate from conns list (#3084)
keep conns_pool of struct connection separate from conns list
and allocate conns list to srv->srvconf.max_conns size at startup

  "Memory fragmentation with HTTP/2 enabled"
2021-09-08 15:06:06 -04:00
Glenn Strauss a8c9b38118 [core] adjust srv->srvconf.max_conns at startup
adjust srv->srvconf.max_conns at startup, if adjustment needed
2021-09-08 15:06:06 -04:00
Glenn Strauss c69450ec04 [core] free fdwaitqueue list when empty
fdwaitqueue is typically unused except in severe overload conditions,
so free up the list storage as soon as it is empty
2021-09-08 15:06:06 -04:00
Glenn Strauss 311d4b0ef4 [core] adjust server overload check
do not re-enable sockets if there are any connections in fdwaitqueue
(server_overload_check() now better mirrors server_load_check())
2021-09-08 15:06:06 -04:00
Glenn Strauss 600dfba3e6 [core] set min srv->max_fds = 32 (sanity check)
(server load checks will fail if value is too low)
2021-09-08 15:06:06 -04:00
Glenn Strauss 6515f77665 [core] clarify error message in gw_backend.c
clarify error message in gw_backend.c if connect() to unix socket fails
2021-09-08 15:06:06 -04:00
Glenn Strauss 8c36615f85 [mod_cgi] cgi.limits "read-timeout" "write-timeout" (#3086)
implement write(), read() timeouts for CGI


  "sockets disabled, out-of-fds with proxy module"
2021-09-08 15:06:06 -04:00
Glenn Strauss da562e3fd6 [core] http_response_read() indicate resp finished
return HANDLER_FINISHED from http_response_read() if response finished,
whether due to reading EOF (prior behavior), or if Content-Length was
provided and we have finished reading Content-Length, or if a module
sets r->resp_body_finished for any other reason.  This may save an
unnecessary poll() and read() to receive EOF when Content-Length has
already been read.
2021-09-08 15:06:06 -04:00
Glenn Strauss 39a577985f [core] improve handling of suboptimal backend wr
more efficiently handle reading of suboptimal backend write behavior

check to accumulate small reads in mem before flushing to temp file
2021-09-08 15:06:06 -04:00
Glenn Strauss 833d658729 [core] http_response_append_{buffer,mem}()
manage r->resp_body_scratchpad in new funcs
rather than
which now only decode chunked encoding, more apropos for the func names
2021-09-08 15:06:06 -04:00
Glenn Strauss 2ef31a1b3f [core] chunkqueue_append_buffer always clears buf
chunkqueue_append_buffer() always clears buffer
(instead of relying on caller to do so after the call)
2021-09-08 15:06:06 -04:00
Glenn Strauss 3f17f94336 [core] tweak buffer merging to reduce mem
tweak buffer merging in chunkqueue_append_buffer() to trade off
a slight increase in copying for slightly more constrained mem usage
2021-09-08 15:06:06 -04:00
Glenn Strauss d958cf3262 [doc] 2021-09-08 15:06:06 -04:00
Glenn Strauss 02646ea2ad [core] connect, write, read timeouts on backends (fixes #3086)
implement connect(), write(), read() timeouts on backends


  "sockets disabled, out-of-fds with proxy module"
2021-09-08 15:06:06 -04:00
Glenn Strauss 4f96dac841 [core] gw_backend_error() shared code 2021-09-04 08:08:27 -04:00
Glenn Strauss ae149c42e5 [core] store time for last r/w to a backend socket 2021-09-04 08:08:26 -04:00
Glenn Strauss 075230158a [core] make missing mod_deflate not a fatal error
mod_compress was removed in lighttpd 1.4.56, subsumed by mod_deflate.

distros may package mod_deflate separately from the lighttpd package.

However, existing configurations may reference mod_compress.
lighttpd maps the reference from mod_compress to mod_deflate,
but after a system is upgraded to lighttpd 1.4.56 or later,
mod_compress may have been removed, and mod_deflate -- which
might be a separate package -- might not be installed.

lighttpd will still issue error trace about the missing mod_deflate
modules, as well as about the unrecognized configuration directives
(compress.* or deflate.*), but this will no longer be a fatal error.
2021-09-04 08:08:26 -04:00