e.g. if /usr/bin/php-cgi does not exist
A distribution package might need to be installed:
'php-cli' Fedora package; 'php7.0-cgi' or 'php5-cgi' Debian package
Fix mod_scgi prefix matching: match the prefix always against url,
not the absolute filepath (regardless of check-local)
(apply fix similar to commit:fe8b7e57 applied to mod_fastcgi in 2008)
setting or removing FD_CLOEXEC flag does not fail
Also the use in mod_fastcgi and mod_scgi is in child after fork().
If the fd already happens to be 0 (should not happen in current code)
and removing the FD_CLOEXEC flag fails, then the backend will fail
to start.
consolidated from CGI, FastCGI, SCGI, SSI
Note: due to prior inconsistencies between the code in mod_cgi,
mod_fastcgi, mod_scgi, and mod_ssi, there are some minor behavior
changes.
CONTENT_LENGTH is now always set, even if 0
(though CONTENT_LENGTH is never set for FASTCGI_AUTHORIZER)
PATH_INFO is created only if present, not if empty.
(mod_fastcgi and mod_ssi previously set PATH_INFO="" (blank value))
PATH_TRANSLATED is now set if PATH_INFO is present
(previously missing from mod_cgi and mod_ssi)
mod_ssi now sets DOCUMENT_ROOT to con->physical.basedir, like others
(previously, mod_ssi set DOCUMENT_ROOT to con->physical.doc_root,
which matched con->physical.basedir unless mod_alias changed basedir)
mod_ssi now sets REQUEST_URI to con->request.orig_uri, like others
(previously, mod_ssi set REQUEST_URI to con->request.uri, which
matched con->request.orig_uri except after redirects, error docs)
Configuring the protocol is controlled with new lighttpd.conf directive:
scgi.protocol = "scgi" # default
scgi.protocol = "uwsgi"
The uwsgi protocol differs from the SCGI protocol only in how the
request is encoded. The response from the backend is handled the
same way for both SCGI and uwsgi protocols.
x-ref: http://uwsgi-docs.readthedocs.io/en/latest/Protocol.html
(for persistence across multiple re-entries into routines upon
receiving fdevent)
(setting module debug flag in global scope is still recommended
since there are places where p->conf.debug is checked since
handler_ctx might not be available at all points)
Due to the POLLHUP behavior triggered on *BSD/Darwin, the shutdown()
had previously been limited to local connections. If interested in
squeezing the last bits of performance out of a machine, an admin
should configure local connections to be AF_UNIX instead of AF_INET
or AF_INET6 to localhost. The reason the shutdown() was originally
added in mod_proxy and mod_scgi was to aggressively reduce the number
of potential sockets in TIME_WAIT held by lighttpd.
(See commit:923688d2 "drain backend socket/pipe bufs upon FDEVENT_HUP",
done for reliability given the aforementioned *BSD/Darwin behavior.)
When using AF_UNIX, the TIME_WAIT issue does not exist, ergo, the
recommendation is to use AF_UNIX for local sockets, when available.
Using AF_UNIX sockets is a better solution to eliminate TIME_WAIT
than is TCP shutdown() half-close which, as we have seen, might not
be handled well by frameworks which are more complex than basic read
request, send response, and close.
x-ref:
"1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin"
https://redmine.lighttpd.net/issues/2743
mod_fastcgi, mod_scgi, and mod_proxy must enforce wait for POLLWR
after EINPROGRESS or else getsockopt(fd, SOL_SOCKET, SO_ERROR, ...)
may succeed even though socket connection is not yet established,
and subsequent writev() will fail ENOTCONN.
(thx pkubaj)
x-ref:
"1.4.40/41 writev failed: Socket is not connected (fastcgi,scgi,proxy)"
https://redmine.lighttpd.net/issues/2744
shutdown(fd, SHUT_WR) after sending request to proxy or SCGI
only if remote is local and platform is not *BSD or Darwin.
The reason this fix is special-casing *BSD and Darwin is that the Single
Unix Specification and POSIX.1-2013 clearly specify that POLLHUP event
should be returned by poll only when the stream is no longer writable.
A half-closed socket that is still writable clearly does not match that
condition, yet that is what I am seeing on Darwin (El Capitan), and
presumably what others are seeing on *BSD, from which Apple originally
inherited the Darwin TCP stack.
Single Unix Specification (SUSv2) from 1997
(yes, that is nearly 20 years ago):
http://pubs.opengroup.org/onlinepubs/007908799/xsh/poll.html
POLLHUP
The device has been disconnected. This event and POLLOUT are
mutually exclusive; a stream can never be writable if a hangup has
occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND or
POLLPRI are not mutually exclusive. This flag is only valid in the
revents bitmask; it is ignored in the events member.
Updated version of The Open Group Base Specifications Issue 7
(published in 2013):
http://pubs.opengroup.org/onlinepubs/9699919799/
POLLHUP
A device has been disconnected, or a pipe or FIFO has been closed
by the last process that had it open for writing. Once set, the
hangup state of a FIFO shall persist until some process opens the
FIFO for writing or until all read-only file descriptors for the
FIFO are closed. This event and POLLOUT are mutually-exclusive;
a stream can never be writable if a hangup has occurred. However,
this event and POLLIN, POLLRDNORM, POLLRDBAND, or POLLPRI are not
mutually-exclusive. This flag is only valid in the revents bitmask;
it shall be ignored in the events member.
x-ref:
"1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin"
https://redmine.lighttpd.net/issues/2743
reverts part of commit:dbdab5db which swapped REQUEST_URI, REDIRECT_URI
x-ref:
"mediawiki redirect loop if REQUEST_URI not orig req in 1.4.40"
https://redmine.lighttpd.net/issues/2738
Explanation:
REQUEST_URI and REDIRECT_URI are not part of CGI standard environment.
The reason for their existence is that PATH_INFO in CGI environment may
be different from the path in the current request. The main reason for
this potential difference is that the URI path is normalized to a path
in the filesystem and tested against the filesystem to determine which
part is SCRIPT_NAME and which part is PATH_INFO. In case-insensitive
filesystems, the URI might be lowercased before testing against the
filesystem, leading to loss of case-sensitive submission in any
resulting PATH_INFO. Also, duplicated slashes "///" and directory
references "/." and "/.." are removed, including prior path component in
the case of "/..". This might be undesirable when the information after
the SCRIPT_NAME is virtual information and there target script needs the
virtual path preserved as-is. In that case, the target script can
re-parse REQUEST_URI (or REDIRECT_URI, as appropriate) to obtain the
unmodified information from the URI.
con->request.uri is equivalent to con->request.orig_uri unless the
request has been internally rewritten (e.g. by mod_rewrite, mod_magnet,
others), in which case con->request.orig_uri is the request made by the
client, and con->request.uri is the current URI being processed.
Historical REQUEST_URI (environment variable) lighttpd inconsistencies
- mod_cml set REQUEST_URI to con->request.orig_uri
- mod_cgi set REQUEST_URI to con->request.orig_uri
- mod_fastcgi set REQUEST_URI to con->request.orig_uri
- mod_scgi set REQUEST_URI to con->request.orig_uri
- mod_ssi set REQUEST_URI to current con->request.uri
- mod_magnet set MAGNET_ENV_REQUEST_URI to current con->request.uri
and MAGNET_ENV_REQUEST_ORIG_URI to con->request.orig_uri
Historical REDIRECT_URI (environment variable) previously set only in
mod_fastcgi and mod_scgi, and set to con->request.uri
Since lighttpd 1.4.40 provides REDIRECT_URI with con->request.orig_uri,
changes were made to REQUEST_URI for consistency, with the hope that
there would be little impact to existing configurations since the
request uri and original request uri are the same unless there has been
an internal redirect. It turns out that various PHP frameworks use
REQUEST_URI and require that it be the original URI requested by client.
Therefore, this change is being reverted, and lighttpd will set
REQUEST_URI to con->request.orig_uri in mod_cgi, mod_fastcgi, mod_scgi
as was done in lighttpd 1.4.39 and earlier. Similarly, REDIRECT_URI
also has the prior behavior in mod_fastcgi and mod_scgi, and added to
mod_cgi.
A future release of lighttpd might change mod_ssi to be consistent with
the other modules in setting REQUEST_URI to con->request.orig_uri and to
add REDIRECT_URI, when an internal redirect has occurred.
Strip bogus "Proxy" header before creating subprocess environment.
(mod_cgi, mod_fastcgi, mod_scgi, mod_ssi, mod_proxy)
Do not emit HTTP_PROXY to subprocess environment.
Some executables use HTTP_PROXY to configure outgoing proxy.
This is not a lighttpd security issue per se, but this change to
lighttpd adds a layer of defense to protect backend processes which
might be vulnerable due to blindly using this untrusted environment
variable. The HTTP_PROXY environment variable should not be trusted
by a program running in a CGI-like environment.
Mitigation in lighttpd <= 1.4.40 is to reject requests w/ Proxy header:
* Create "/path/to/deny-proxy.lua", read-only to lighttpd, with content:
if (lighty.request["Proxy"] == nil) then return 0 else return 403 end
* Modify lighttpd.conf to load mod_magnet and run lua code
server.modules += ( "mod_magnet" )
magnet.attract-raw-url-to = ( "/path/to/deny-proxy.lua" )
References:
https://www.kb.cert.org/vuls/id/797896
CGI web servers assign Proxy header values from client requests to
internal HTTP_PROXY environment variables
https://httpoxy.org/
httpoxy: A CGI application vulnerability
error out if duplicated unix socket path and different bin-path is
used by lighttpd to start the backend. This is an error.
Updates commits 97556d99 and b67ff2aa
x-ref:
"Should warn if two FastCGI servers point to same socket file"
https://redmine.lighttpd.net/issues/319
silently use same 'host' if unix socket path is duplicated
(instead of erroring out). Updates commit 97556d99
(It is possible for use of += in config to result in duplicated
fastcgi.server and scgi.server entries)
x-ref:
"Should warn if two FastCGI servers point to same socket file"
https://redmine.lighttpd.net/issues/319
do not expose server info server_tag is configured by admin
default con->conf.server_tag is still PACKAGE_DESC, those
admin can configure server.tag otherwise.
(these changes reduce recompilation and relinking when switching
brances in source control)
cygwin does not support ioctl on sockets, returning EOPTNOTSUPP
(would be better if cygwin used Windows ioctlsocket() instead)
Windows uses signed (socklen_t), so add some casts to quiet warnings
Windows path handling is convoluted, so disable one tests in mod_fastcgi
since trailing spaces are removed from URL for _WIN32 and __CYGWIN__ in
response.c
Set server.stream-response-body = 1 or server.stream-response-body = 2
to have lighttpd stream response body to client as it arrives from the
backend (CGI, FastCGI, SCGI, proxy).
default: buffer entire response body before sending response to client.
(This preserves existing behavior for now, but may in the future be
changed to stream response to client, which is the behavior more
commonly expected.)
x-ref:
"fastcgi, cgi, flush, php5 problem."
https://redmine.lighttpd.net/issues/949
"Random crashing on FreeBSD 6.1"
https://redmine.lighttpd.net/issues/760
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283
"lighttpd+fastcgi memory problem"
https://redmine.lighttpd.net/issues/1387
move code in dynamic handlers (CGI, FastCGI, SCGI, proxy)
from *_handle_fdevent() to *_recv_response() for reuse
outside the *_handle_fdevent() routine
Set server.stream-request-body = 1 or server.stream-request-body = 2
to have lighttpd connect to backend (CGI, FastCGI, SCGI, proxy)
immediately after parsing request headers, and to stream request body
as it arrives.
default: buffer entire request body before connecting to backend,
in order to avoid tying up (limited) backend resources which are often
implemented using libraries which wait for entire request body before
proceeding.
x-ref:
"Reimplement upload (POST) handling to match apache/zeus/thttpd/boa functionality"
https://redmine.lighttpd.net/issues/376
server.error-handler preserves HTTP status error code when error page
is static, and allows dynamic handlers to change HTTP status code
when error page is provided by dynamic handler. server.error-handler
intercepts all HTTP status codes >= 400 except when the content is
generated by a dynamic handler (cgi, ssi, fastcgi, scgi, proxy, lua).
The request method is unconditionally changed to GET for the request
to service the error handler, and the original request method is
later restored (for logging purposes). request body from the
original request, if present, is discarded.
server.error-handler is somewhat similar to server.error-handler-404,
but server.error-handler-404 is now deprecated, intercepts only 404
and 403 HTTP status codes, and returns 200 OK for static error pages,
a source of confusion for some admins. On the other hand, the new
server.error-handler, when set, will intercept all HTTP status error
codes >= 400. server.error-handler takes precedence over
server.error-handler-404 when both are set.
NOTE: a major difference between server.error-handler and the
now-deprecated server.error-handler-404 is that the values of the
non-standard CGI environment variables REQUEST_URI and REDIRECT_URI
have been swapped. Since REDIRECT_STATUS is the original HTTP
status code, REDIRECT_URI is now the original request, and REQUEST_URI
is the current request (e.g. the URI/URL to the error handler).
The prior behavior -- which reversed REQUEST_URI and REDIRECT_URI values
from those described above -- is preserved for server.error-handler-404.
Additionally, REDIRECT_STATUS is now available to mod_magnet, which
continues to have access to request.uri and request.orig_uri.
See further discussion at https://redmine.lighttpd.net/issues/2702
and https://redmine.lighttpd.net/issues/1828
github: closes #36
set REDIRECT_STATUS to con->error_handler_saved_status in dynamic
handlers for PHP compiled with --force-redirect. Set to "200"
if (0 == con->error_handler_saved_status)
(mod_cgi, mod_fastcgi, mod_scgi, mod_ssi)
FYI: setting REDIRECT_STATUS in con->environment allows access and
manipulation by mod_magnet.
x-ref:
"REDIRECT_STATUS == 200 on 404 redirect"
https://redmine.lighttpd.net/issues/1828
github: closes #35
iterate over environ via array-index notation with char **ptr on stack
(instead of repeatedly re-accessing global 'environ')
check getsockname() return values including addrlen
[mod_dirlisting] pass buf size into http_list_directory_sizefmt()
github: resolves #48
dynamic handlers mod_fastcgi, mod_scgi, and mod_proxy can now read
response from backend prior to finishing sending request body.
If the backend closes the connections (or shuts down socket write
end so that lighttpd read() 0 to indicate EOF), then lighttpd will
abort attempting to send request body to backend.
x-ref:
"mod_fastcgi should handle "quick" responses"
https://redmine.lighttpd.net/issues/2566
"FastCGI FCGI_STDOUT before FCGI_STDIN bug"
https://redmine.lighttpd.net/issues/131
remove handle_joblist hook and remove the hooks defined in
mod_fastcgi and mod_scgi. The calls made to fdevent management
are redundant. If the calls were actually needed, then
mod_proxy would have needed a handle_joblist handler, too.
to detect client disconnect. Do so even when waiting on backend,
and not polling for POLLRD or POLLWR on client connection.
This reduces unnecessary load on backends when backends are slow
to respond and client has given up waiting.
x-ref:
"https://redmine.lighttpd.net/issues/399"
FastCGI performance on high load
read request body in dynamic handlers supporting request body
(mod_cgi, mod_fastcgi, mod_proxy, mod_scgi, mod_webdav)
(In the future, each dynamic handler might choose whether or not to
buffer request body or to stream request body to backend as request
body is received.)
modify mod_webdav to mark request in handle_physical hook, and move
the main logic to handle_subrequest hook, where the main logic is
for other dynamic handlers.
read request body right before calling subrequest handler,
allowing request to be handled prior to reading request body,
e.g. to send 401 Unauthorized response when authentication is required
(In the future, this might move into each dynamic handler which supports
request body (mod_cgi, mod_fastcgi, mod_proxy, mod_scgi, mod_webdav) so
that each dynamic handler can choose whether or not to buffer request
body or to stream request body to backend as request body is received.)
keep-alive is disabled if request body has not been completely read
prior to sending response
x-ref:
"HTTP 401 Unauthorized only sent back after full POST request is read"
https://redmine.lighttpd.net/issues/2541
for dynamic handlers mod_cgi, mod_fastcgi, mod_scgi, mod_proxy
(mod_cgi control flow logic simplification began in a prior commit)
- connection state machine calls the subrequest handler
- subrequest handler sets up the connection to the backend
and registers fdevent handler to handle backend events
- fdevent handler handles backend events and then schedules
a call to connection state machine
- when retrying an alternate backend, backend state is reset
and then response state is reset so that the connection state
machine will call back into the subrequest handler to retry
Dynamic handlers no longer directly modify connection state
(calls to connection_set_state() from dynamic handlers were removed)
Dynamic handlers no longer reset con->physical.path, and they
preserve con->mode when retrying alternate backends. This is done
to skip repeated processing in response.c:http_response_prepare()
While this patch increases consistency in control flow handling,
there is more work to be done that can further improve upon this.
x-ref:
"handle-req time too long"
https://redmine.lighttpd.net/issues/1149
See doc/config/lighttpd.conf for explanation of listen() backlog queue
Additionally, mod_fastcgi and mod_scgi backend servers can now also be
configured with separate listen-backlog settings per server
x-ref:
"add server.listen-backlog option instead of hard-coded value (128 * 8) for listen()"
https://redmine.lighttpd.net/issues/2116
"Don't disable backend when overloaded"
https://redmine.lighttpd.net/issues/1825
github:
Closes #50
error if unix socket path is duplicated
(does not check across modules, but will detect duplicated unix socket
paths within fastcgi directives, and separately, duplicated unix socket
paths within scgi directives)
When a sockaddr_un, sockaddr_in or sockaddr_in6 structure
is allocated on the stack or heap, it may contain random
byte values.
The "unused" and "reserved" parts must be zerod otherwise
unexpected failures may occur. The simplest way to do
this and be compatible with various platforms' struct
layouts is just to memset them to 0.
Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3059 152afb58-edef-0310-8abb-c4023f1b3aa9
Kyle J. McKay