lighttpd 1.4.x https://www.lighttpd.net/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2193 lines
74 KiB

/* mod_deflate
*
*
* bug fix on Robert Jakabosky from alphatrade.com's lighttp 1.4.10 mod_deflate patch
*
* Bug fix and new features:
* 1) fix loop bug when content-length is bigger than work-block-size*k
*
* -------
*
* lighttpd-1.4.26.mod_deflate.patch from
* https://redmine.lighttpd.net/projects/1/wiki/Docs_ModDeflate
*
* -------
*
* Patch further modified in this incarnation.
*
* Note: this patch only handles completed responses
* (r->resp_body_finished)
* this patch does not currently handle streaming dynamic responses,
* and therefore also does not worry about Transfer-Encoding: chunked
* (or having separate con->output_queue for chunked-encoded output)
* (or using separate buffers per connection instead of p->tmp_buf)
* (or handling interactions with block buffering and write timeouts)
*
* Bug fix:
* - fixed major bug with compressing chunks with offset > 0
* x-ref:
* "Response breaking in mod_deflate"
* https://redmine.lighttpd.net/issues/986
* - fix broken (in some cases) chunk accounting in deflate_compress_response()
* - fix broken bzip2
* x-ref:
* "mod_deflate's bzip2 broken by default"
* https://redmine.lighttpd.net/issues/2035
* - fix mismatch with current chunk interfaces
* x-ref:
* "Weird things in chunk.c (functions only handling specific cases, unexpected behaviour)"
* https://redmine.lighttpd.net/issues/1510
*
* Behavior changes from prior patch:
* - deflate.mimetypes must now be configured to enable compression
* deflate.mimetypes = ( ) # compress nothing (disabled; default)
* deflate.mimetypes = ( "" ) # compress all mimetypes
* deflate.mimetypes = ( "text/" ) # compress text/... mimetypes
* x-ref:
* "mod_deflate enabled by default"
* https://redmine.lighttpd.net/issues/1394
* - deflate.enabled directive removed (see new behavior of deflate.mimetypes)
* - deflate.debug removed (was developer debug trace, not end-user debug)
* - deflate.bzip2 replaced with deflate.allowed-encodings (like mod_compress)
* x-ref:
* "mod_deflate should allow limiting of compression algorithm from the configuration file"
* https://redmine.lighttpd.net/issues/996
* "mod_compress disabling methods"
* https://redmine.lighttpd.net/issues/1773
* - deflate.nocompress-url removed since disabling compression for a URL
* can now easily be done by setting to a blank list either directive
* deflate.accept_encodings = () or deflate.mimetypes = () in a conditional
* block, e.g. $HTTP["url"] =~ "....." { deflate.mimetypes = ( ) }
* - deflate.sync-flush removed; controlled by r->conf.stream_response_body
* (though streaming compression not currently implemented in mod_deflate)
* - inactive directives in this patch
* (since r->resp_body_finished required)
* deflate.work-block-size
* deflate.output-buffer-size
* - remove weak file size check; SIGBUS is trapped, file that shrink will error
* x-ref:
* "mod_deflate: filesize check is too weak"
* https://redmine.lighttpd.net/issues/1512
* - change default deflate.min-compress-size from 0 to now be 256
* http://webmasters.stackexchange.com/questions/31750/what-is-recommended-minimum-object-size-for-gzip-performance-benefits
* Apache 2.4 mod_deflate minimum is 68 bytes
* Akamai recommends minimum 860 bytes
* Google recommends minimum be somewhere in range between 150 and 1024 bytes
* - deflate.max-compress-size new directive (in kb like compress.max_filesize)
* - deflate.mem-level removed (too many knobs for little benefit)
* - deflate.window-size removed (too many knobs for little benefit)
*
* Future:
* - config directives may be changed, renamed, or removed
* e.g. A set of reasonable defaults might be chosen
* instead of making them configurable.
* deflate.min-compress-size
* - might add deflate.mimetypes-exclude = ( ... ) for list of mimetypes
* to avoid compressing, even if a broader deflate.mimetypes matched,
* e.g. to compress all "text/" except "text/special".
*
* Implementation notes:
* - http_chunk_append_mem() used instead of http_chunk_append_buffer()
* so that p->tmp_buf can be large and re-used. This results in an extra copy
* of compressed data before data is sent to network, though if the compressed
* size is larger than 64k, it ends up being sent to a temporary file on
* disk without suffering an extra copy in memory, and without extra chunk
* create and destroy. If this is ever changed to give away buffers, then use
* a unique hctx->output buffer per hctx; do not reuse p->tmp_buf across
* multiple requests being handled in parallel.
*/
#include "first.h"
#include <sys/types.h>
#include <sys/stat.h>
#include "sys-mmap.h"
#ifdef HAVE_MMAP
#include "sys-setjmp.h"
#endif
#include "sys-time.h"
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h> /* getpid() read() unlink() write() */
#include "base.h"
#include "ck.h"
#include "fdevent.h"
#include "log.h"
#include "buffer.h"
#include "http_chunk.h"
#include "http_etag.h"
#include "http_header.h"
#include "response.h"
#include "stat_cache.h"
#include "plugin.h"
#if defined HAVE_ZLIB_H && defined HAVE_LIBZ
# define USE_ZLIB
# include <zlib.h>
#endif
#ifndef Z_DEFAULT_COMPRESSION
#define Z_DEFAULT_COMPRESSION -1
#endif
#ifndef MAX_WBITS
#define MAX_WBITS 15
#endif
#if defined HAVE_BZLIB_H && defined HAVE_LIBBZ2
# define USE_BZ2LIB
/* we don't need stdio interface */
# define BZ_NO_STDIO
# include <bzlib.h>
#endif
#if defined HAVE_BROTLI_ENCODE_H && defined HAVE_BROTLI
# define USE_BROTLI
# include <brotli/encode.h>
#endif
#if defined HAVE_ZSTD_H && defined HAVE_ZSTD
# define USE_ZSTD
# include <zstd.h>
#endif
#ifndef HAVE_LIBZ
#undef HAVE_LIBDEFLATE
#endif
/* request: accept-encoding */
#define HTTP_ACCEPT_ENCODING_IDENTITY BV(0)
#define HTTP_ACCEPT_ENCODING_GZIP BV(1)
#define HTTP_ACCEPT_ENCODING_DEFLATE BV(2)
#define HTTP_ACCEPT_ENCODING_COMPRESS BV(3)
#define HTTP_ACCEPT_ENCODING_BZIP2 BV(4)
#define HTTP_ACCEPT_ENCODING_X_GZIP BV(5)
#define HTTP_ACCEPT_ENCODING_X_BZIP2 BV(6)
#define HTTP_ACCEPT_ENCODING_BR BV(7)
#define HTTP_ACCEPT_ENCODING_ZSTD BV(8)
typedef struct {
struct {
int clevel; /*(compression level)*/
int windowBits;
int memLevel;
int strategy;
} gzip;
struct {
uint32_t quality; /*(compression level)*/
uint32_t window;
uint32_t mode;
} brotli;
struct {
int clevel; /*(compression level)*/
int strategy;
int windowLog;
} zstd;
struct {
int clevel; /*(compression level)*/
} bzip2;
} encparms;
typedef struct {
const array *mimetypes;
const buffer *cache_dir;
unsigned int max_compress_size;
unsigned short min_compress_size;
unsigned short output_buffer_size;
unsigned short work_block_size;
unsigned short sync_flush;
short compression_level;
uint16_t * allowed_encodings;
double max_loadavg;
const encparms *params;
} plugin_config;
typedef struct {
PLUGIN_DATA;
plugin_config defaults;
plugin_config conf;
buffer tmp_buf;
} plugin_data;
typedef struct {
union {
#ifdef USE_ZLIB
z_stream z;
#endif
#ifdef USE_BZ2LIB
bz_stream bz;
#endif
#ifdef USE_BROTLI
BrotliEncoderState *br;
#endif
#ifdef USE_ZSTD
ZSTD_CStream *cctx;
#endif
int dummy;
} u;
off_t bytes_in;
off_t bytes_out;
buffer *output;
plugin_data *plugin_data;
request_st *r;
int compression_type;
int cache_fd;
char *cache_fn;
chunkqueue in_queue;
} handler_ctx;
static handler_ctx *handler_ctx_init() {
handler_ctx *hctx;
hctx = calloc(1, sizeof(*hctx));
chunkqueue_init(&hctx->in_queue);
hctx->cache_fd = -1;
return hctx;
}
static void handler_ctx_free(handler_ctx *hctx) {
if (hctx->cache_fn) {
unlink(hctx->cache_fn);
free(hctx->cache_fn);
}
if (-1 != hctx->cache_fd)
close(hctx->cache_fd);
#if 0
if (hctx->output != &p->tmp_buf) {
buffer_free(hctx->output);
}
#endif
chunkqueue_reset(&hctx->in_queue);
free(hctx);
}
INIT_FUNC(mod_deflate_init) {
plugin_data * const p = calloc(1, sizeof(plugin_data));
#ifdef USE_ZSTD
buffer_string_prepare_copy(&p->tmp_buf, ZSTD_CStreamOutSize());
#else
buffer_string_prepare_copy(&p->tmp_buf, 65536);
#endif
return p;
}
FREE_FUNC(mod_deflate_free) {
plugin_data *p = p_d;
free(p->tmp_buf.ptr);
if (NULL == p->cvlist) return;
/* (init i to 0 if global context; to 1 to skip empty global context) */
for (int i = !p->cvlist[0].v.u2[1], used = p->nconfig; i < used; ++i) {
config_plugin_value_t *cpv = p->cvlist + p->cvlist[i].v.u2[0];
for (; -1 != cpv->k_id; ++cpv) {
if (cpv->vtype != T_CONFIG_LOCAL || NULL == cpv->v.v) continue;
switch (cpv->k_id) {
case 1: /* deflate.allowed-encodings */
case 14:/* deflate.params */
free(cpv->v.v);
break;
default:
break;
}
}
}
}
#if defined(_WIN32) && !defined(__CYGWIN__)
#define mkdir(x,y) mkdir(x)
#endif
static int mkdir_for_file (char *fn) {
for (char *p = fn; (p = strchr(p + 1, '/')) != NULL; ) {
if (p[1] == '\0') return 0; /* ignore trailing slash */
*p = '\0';
int rc = mkdir(fn, 0700);
*p = '/';
if (0 != rc && errno != EEXIST) return -1;
}
return 0;
}
static int mkdir_recursive (char *dir) {
return 0 == mkdir_for_file(dir) && (0 == mkdir(dir,0700) || errno == EEXIST)
? 0
: -1;
}
static buffer * mod_deflate_cache_file_name(request_st * const r, const buffer *cache_dir, const buffer * const etag) {
/* XXX: future: for shorter paths into the cache, we could checksum path,
* and then shard it to avoid a huge single directory.
* Alternatively, could use &r->uri.path, minus any
* (matching) &r->pathinfo suffix, with result url-encoded
* Alternative, we could shard etag which is already our "checksum" */
buffer * const tb = r->tmp_buf;
[multiple] reduce redundant NULL buffer checks This commit is a large set of code changes and results in removal of hundreds, perhaps thousands, of CPU instructions, a portion of which are on hot code paths. Most (buffer *) used by lighttpd are not NULL, especially since buffers were inlined into numerous larger structs such as request_st and chunk. In the small number of instances where that is not the case, a NULL check is often performed earlier in a function where that buffer is later used with a buffer_* func. In the handful of cases that remained, a NULL check was added, e.g. with r->http_host and r->conf.server_tag. - check for empty strings at config time and set value to NULL if blank string will be ignored at runtime; at runtime, simple pointer check for NULL can be used to check for a value that has been set and is not blank ("") - use buffer_is_blank() instead of buffer_string_is_empty(), and use buffer_is_unset() instead of buffer_is_empty(), where buffer is known not to be NULL so that NULL check can be skipped - use buffer_clen() instead of buffer_string_length() when buffer is known not to be NULL (to avoid NULL check at runtime) - use buffer_truncate() instead of buffer_string_set_length() to truncate string, and use buffer_extend() to extend Examples where buffer known not to be NULL: - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL (though we might set it to NULL if buffer_is_blank(cpv->v.b)) - address of buffer is arg (&foo) (compiler optimizer detects this in most, but not all, cases) - buffer is checked for NULL earlier in func - buffer is accessed in same scope without a NULL check (e.g. b->ptr) internal behavior change: callers must not pass a NULL buffer to some funcs. - buffer_init_buffer() requires non-null args - buffer_copy_buffer() requires non-null args - buffer_append_string_buffer() requires non-null args - buffer_string_space() requires non-null arg
12 months ago
buffer_copy_path_len2(tb, BUF_PTR_LEN(cache_dir),
BUF_PTR_LEN(&r->physical.path));
buffer_append_str2(tb, CONST_STR_LEN("-"), /*(strip surrounding '"')*/
[multiple] reduce redundant NULL buffer checks This commit is a large set of code changes and results in removal of hundreds, perhaps thousands, of CPU instructions, a portion of which are on hot code paths. Most (buffer *) used by lighttpd are not NULL, especially since buffers were inlined into numerous larger structs such as request_st and chunk. In the small number of instances where that is not the case, a NULL check is often performed earlier in a function where that buffer is later used with a buffer_* func. In the handful of cases that remained, a NULL check was added, e.g. with r->http_host and r->conf.server_tag. - check for empty strings at config time and set value to NULL if blank string will be ignored at runtime; at runtime, simple pointer check for NULL can be used to check for a value that has been set and is not blank ("") - use buffer_is_blank() instead of buffer_string_is_empty(), and use buffer_is_unset() instead of buffer_is_empty(), where buffer is known not to be NULL so that NULL check can be skipped - use buffer_clen() instead of buffer_string_length() when buffer is known not to be NULL (to avoid NULL check at runtime) - use buffer_truncate() instead of buffer_string_set_length() to truncate string, and use buffer_extend() to extend Examples where buffer known not to be NULL: - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL (though we might set it to NULL if buffer_is_blank(cpv->v.b)) - address of buffer is arg (&foo) (compiler optimizer detects this in most, but not all, cases) - buffer is checked for NULL earlier in func - buffer is accessed in same scope without a NULL check (e.g. b->ptr) internal behavior change: callers must not pass a NULL buffer to some funcs. - buffer_init_buffer() requires non-null args - buffer_copy_buffer() requires non-null args - buffer_append_string_buffer() requires non-null args - buffer_string_space() requires non-null arg
12 months ago
etag->ptr+1, buffer_clen(etag)-2);
return tb;
}
static void mod_deflate_cache_file_open (handler_ctx * const hctx, const buffer * const fn) {
/* race exists whereby up to # workers might attempt to compress same
* file at same time if requested at same time, but this is unlikely
* and resolves itself by atomic rename into place when done */
[multiple] reduce redundant NULL buffer checks This commit is a large set of code changes and results in removal of hundreds, perhaps thousands, of CPU instructions, a portion of which are on hot code paths. Most (buffer *) used by lighttpd are not NULL, especially since buffers were inlined into numerous larger structs such as request_st and chunk. In the small number of instances where that is not the case, a NULL check is often performed earlier in a function where that buffer is later used with a buffer_* func. In the handful of cases that remained, a NULL check was added, e.g. with r->http_host and r->conf.server_tag. - check for empty strings at config time and set value to NULL if blank string will be ignored at runtime; at runtime, simple pointer check for NULL can be used to check for a value that has been set and is not blank ("") - use buffer_is_blank() instead of buffer_string_is_empty(), and use buffer_is_unset() instead of buffer_is_empty(), where buffer is known not to be NULL so that NULL check can be skipped - use buffer_clen() instead of buffer_string_length() when buffer is known not to be NULL (to avoid NULL check at runtime) - use buffer_truncate() instead of buffer_string_set_length() to truncate string, and use buffer_extend() to extend Examples where buffer known not to be NULL: - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL (though we might set it to NULL if buffer_is_blank(cpv->v.b)) - address of buffer is arg (&foo) (compiler optimizer detects this in most, but not all, cases) - buffer is checked for NULL earlier in func - buffer is accessed in same scope without a NULL check (e.g. b->ptr) internal behavior change: callers must not pass a NULL buffer to some funcs. - buffer_init_buffer() requires non-null args - buffer_copy_buffer() requires non-null args - buffer_append_string_buffer() requires non-null args - buffer_string_space() requires non-null arg
12 months ago
const uint32_t fnlen = buffer_clen(fn);
hctx->cache_fn = malloc(fnlen+1+LI_ITOSTRING_LENGTH+1);
force_assert(hctx->cache_fn);
memcpy(hctx->cache_fn, fn->ptr, fnlen);
hctx->cache_fn[fnlen] = '.';
const size_t ilen =
li_itostrn(hctx->cache_fn+fnlen+1, LI_ITOSTRING_LENGTH, getpid());
hctx->cache_fn[fnlen+1+ilen] = '\0';
hctx->cache_fd = fdevent_open_cloexec(hctx->cache_fn, 1, O_RDWR|O_CREAT, 0600);
if (-1 == hctx->cache_fd) {
free(hctx->cache_fn);
hctx->cache_fn = NULL;
}
}
static int mod_deflate_cache_file_finish (request_st * const r, handler_ctx * const hctx, const buffer * const fn) {
if (0 != fdevent_rename(hctx->cache_fn, fn->ptr))
return -1;
free(hctx->cache_fn);
hctx->cache_fn = NULL;
chunkqueue_reset(&r->write_queue);
int rc = http_chunk_append_file_fd(r, fn, hctx->cache_fd, hctx->bytes_out);
hctx->cache_fd = -1;
return rc;
}
static void mod_deflate_merge_config_cpv(plugin_config * const pconf, const config_plugin_value_t * const cpv) {
switch (cpv->k_id) { /* index into static config_plugin_keys_t cpk[] */
case 0: /* deflate.mimetypes */
pconf->mimetypes = cpv->v.a;
break;
case 1: /* deflate.allowed-encodings */
if (cpv->vtype == T_CONFIG_LOCAL)
pconf->allowed_encodings = cpv->v.v;
break;
case 2: /* deflate.max-compress-size */
pconf->max_compress_size = cpv->v.u;
break;
case 3: /* deflate.min-compress-size */
pconf->min_compress_size = cpv->v.shrt;
break;
case 4: /* deflate.compression-level */
pconf->compression_level = (short)cpv->v.shrt;
break;
case 5: /* deflate.output-buffer-size */
pconf->output_buffer_size = cpv->v.shrt;
break;
case 6: /* deflate.work-block-size */
pconf->work_block_size = cpv->v.shrt;
break;
case 7: /* deflate.max-loadavg */
pconf->max_loadavg = cpv->v.d;
break;
case 8: /* deflate.cache-dir */
pconf->cache_dir = cpv->v.b;
break;
#if 0 /*(cpv->k_id remapped in mod_deflate_set_defaults())*/
case 9: /* compress.filetype */
pconf->mimetypes = cpv->v.a;
break;
case 10:/* compress.allowed-encodings */
if (cpv->vtype == T_CONFIG_LOCAL)
pconf->allowed_encodings = cpv->v.v;
break;
case 11:/* compress.cache-dir */
pconf->cache_dir = cpv->v.b;
break;
case 12:/* compress.max-filesize */
pconf->max_compress_size = cpv->v.u;
break;
case 13:/* compress.max-loadavg */
pconf->max_loadavg = cpv->v.d;
break;
#endif
case 14:/* deflate.params */
if (cpv->vtype == T_CONFIG_LOCAL)
pconf->params = cpv->v.v;
break;
default:/* should not happen */
return;
}
}
static void mod_deflate_merge_config(plugin_config * const pconf, const config_plugin_value_t *cpv) {
do {
mod_deflate_merge_config_cpv(pconf, cpv);
} while ((++cpv)->k_id != -1);
}
static void mod_deflate_patch_config(request_st * const r, plugin_data * const p) {
memcpy(&p->conf, &p->defaults, sizeof(plugin_config));
for (int i = 1, used = p->nconfig; i < used; ++i) {
if (config_check_cond(r, (uint32_t)p->cvlist[i].k_id))
mod_deflate_merge_config(&p->conf, p->cvlist + p->cvlist[i].v.u2[0]);
}
}
static encparms * mod_deflate_parse_params(const array * const a, log_error_st * const errh) {
encparms * params = calloc(1, sizeof(encparms));
force_assert(params);
/* set defaults */
#ifdef USE_ZLIB
params->gzip.clevel = 0; /*(unset)*/
params->gzip.windowBits = MAX_WBITS;
params->gzip.memLevel = 8;
params->gzip.strategy = Z_DEFAULT_STRATEGY;
#endif
#ifdef USE_BROTLI
/* BROTLI_DEFAULT_QUALITY is 11 and can be *very* time-consuming */
params->brotli.quality = 5;
params->brotli.window = BROTLI_DEFAULT_WINDOW;
params->brotli.mode = BROTLI_MODE_GENERIC;
#endif
#ifdef USE_ZSTD
params->zstd.clevel = ZSTD_CLEVEL_DEFAULT;
params->zstd.strategy = 0; /*(use default strategy)*/
params->zstd.windowLog = 0;/*(use default windowLog)*/
#endif
#ifdef USE_BZ2LIB
params->bzip2.clevel = 0; /*(unset)*/
#endif
for (uint32_t i = 0; i < a->used; ++i) {
const data_unset * const du = a->data[i];
#if defined(USE_ZLIB) || defined(USE_BZ2LIB) || defined(USE_BROTLI) \
|| defined(USE_ZSTD)
int32_t v = config_plugin_value_to_int32(du, -1);
#endif
#ifdef USE_BROTLI
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("BROTLI_PARAM_QUALITY"))) {
/*(future: could check for string and then look for and translate
* BROTLI_DEFAULT_QUALITY BROTLI_MIN_QUALITY BROTLI_MAX_QUALITY)*/
if (BROTLI_MIN_QUALITY <= v && v <= BROTLI_MAX_QUALITY)
params->brotli.quality = (uint32_t)v; /* 0 .. 11 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for BROTLI_PARAM_QUALITY");
continue;
}
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("BROTLI_PARAM_LGWIN"))) {
/*(future: could check for string and then look for and translate
* BROTLI_DEFAULT_WINDOW
* BROTLI_MIN_WINDOW_BITS BROTLI_MAX_WINDOW_BITS)*/
if (BROTLI_MIN_WINDOW_BITS <= v && v <= BROTLI_MAX_WINDOW_BITS)
params->brotli.window = (uint32_t)v; /* 10 .. 24 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for BROTLI_PARAM_LGWIN");
continue;
}
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("BROTLI_PARAM_MODE"))) {
/*(future: could check for string and then look for and translate
* BROTLI_MODE_GENERIC BROTLI_MODE_TEXT BROTLI_MODE_FONT)*/
if (0 <= v && v <= 2)
params->brotli.mode = (uint32_t)v; /* 0 .. 2 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for BROTLI_PARAM_MODE");
continue;
}
#endif
#ifdef USE_ZSTD
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("ZSTD_c_compressionLevel"))) {
params->zstd.clevel = v;
/*(not warning if number parse error. future: to detect, could
* use absurd default to config_plugin_value_to_int32 to detect)*/
continue;
}
#if ZSTD_VERSION_NUMBER >= 10000+400+0 /* v1.4.0 */
/*(XXX: (selected) experimental API params in zstd v1.4.0)*/
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("ZSTD_c_strategy"))) {
/*(future: could check for string and then look for and translate
* enum ZSTD_strategy ZSTD_STRATEGY_MIN ZSTD_STRATEGY_MAX)*/
#ifndef ZSTD_STRATEGY_MIN
#define ZSTD_STRATEGY_MIN 1
#endif
#ifndef ZSTD_STRATEGY_MAX
#define ZSTD_STRATEGY_MAX 9
#endif
if (ZSTD_STRATEGY_MIN <= v && v <= ZSTD_STRATEGY_MAX)
params->zstd.strategy = v; /* 1 .. 9 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for ZSTD_c_strategy");
continue;
}
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("ZSTD_c_windowLog"))) {
/*(future: could check for string and then look for and translate
* ZSTD_WINDOWLOG_MIN ZSTD_WINDOWLOG_MAX)*/
#ifndef ZSTD_WINDOWLOG_MIN
#define ZSTD_WINDOWLOG_MIN 10
#endif
#ifndef ZSTD_WINDOWLOG_MAX_32
#define ZSTD_WINDOWLOG_MAX_32 30
#endif
#ifndef ZSTD_WINDOWLOG_MAX_64
#define ZSTD_WINDOWLOG_MAX_64 31
#endif
#ifndef ZSTD_WINDOWLOG_MAX
#define ZSTD_WINDOWLOG_MAX \
(sizeof(size_t)==4 ? ZSTD_WINDOWLOG_MAX_32 : ZSTD_WINDOWLOG_MAX_64)
#endif
if (ZSTD_WINDOWLOG_MIN <= v && v <= ZSTD_WINDOWLOG_MAX)
params->zstd.windowLog = v;/* 10 .. 31 *//*(30 max for 32-bit)*/
else
log_error(errh, __FILE__, __LINE__,
"invalid value for ZSTD_c_windowLog");
continue;
}
#endif
#endif
#ifdef USE_ZLIB
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("gzip.level"))) {
if (1 <= v && v <= 9)
params->gzip.clevel = v; /* 1 .. 9 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for gzip.level");
continue;
}
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("gzip.windowBits"))) {
if (9 <= v && v <= 15)
params->gzip.windowBits = v; /* 9 .. 15 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for gzip.windowBits");
continue;
}
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("gzip.memLevel"))) {
if (1 <= v && v <= 9)
params->gzip.memLevel = v; /* 1 .. 9 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for gzip.memLevel");
continue;
}
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("gzip.strategy"))) {
/*(future: could check for string and then look for and translate
* Z_DEFAULT_STRATEGY Z_FILTERED Z_HUFFMAN_ONLY Z_RLE Z_FIXED)*/
if (0 <= v && v <= 4)
params->gzip.strategy = v; /* 0 .. 4 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for gzip.strategy");
continue;
}
#endif
#ifdef USE_BZ2LIB
if (buffer_eq_icase_slen(&du->key,
CONST_STR_LEN("bzip2.blockSize100k"))) {
if (1 <= v && v <= 9) /*(bzip2 blockSize100k param)*/
params->bzip2.clevel = v; /* 1 .. 9 */
else
log_error(errh, __FILE__, __LINE__,
"invalid value for bzip2.blockSize100k");
continue;
}
#endif
log_error(errh, __FILE__, __LINE__,
"unrecognized param: %s", du->key.ptr);
}
return params;
}
static uint16_t * mod_deflate_encodings_to_flags(const array *encodings) {
if (encodings->used) {
uint16_t * const x = calloc(encodings->used+1, sizeof(short));
force_assert(x);
int i = 0;
for (uint32_t j = 0; j < encodings->used; ++j) {
#if defined(USE_ZLIB) || defined(USE_BZ2LIB) || defined(USE_BROTLI) \
|| defined(USE_ZSTD)
data_string *ds = (data_string *)encodings->data[j];
#endif
#ifdef USE_ZLIB /* "gzip", "x-gzip" */
if (NULL != strstr(ds->value.ptr, "gzip"))
x[i++] = HTTP_ACCEPT_ENCODING_GZIP
| HTTP_ACCEPT_ENCODING_X_GZIP;
if (NULL != strstr(ds->value.ptr, "deflate"))
x[i++] = HTTP_ACCEPT_ENCODING_DEFLATE;
/*
if (NULL != strstr(ds->value.ptr, "compress"))
x[i++] = HTTP_ACCEPT_ENCODING_COMPRESS;
*/
#endif
#ifdef USE_BZ2LIB /* "bzip2", "x-bzip2" */
if (NULL != strstr(ds->value.ptr, "bzip2"))
x[i++] = HTTP_ACCEPT_ENCODING_BZIP2
| HTTP_ACCEPT_ENCODING_X_BZIP2;
#endif
#ifdef USE_BROTLI /* "br" (also accepts "brotli") */
if (NULL != strstr(ds->value.ptr, "br"))
x[i++] = HTTP_ACCEPT_ENCODING_BR;
#endif
#ifdef USE_ZSTD
if (NULL != strstr(ds->value.ptr, "zstd"))
x[i++] = HTTP_ACCEPT_ENCODING_ZSTD;
#endif
}
x[i] = 0; /* end of list */
return x;
}
else {
/* default encodings */
uint16_t * const x = calloc(4+1, sizeof(short));
force_assert(x);
int i = 0;
#ifdef USE_ZSTD
x[i++] = HTTP_ACCEPT_ENCODING_ZSTD;
#endif
#ifdef USE_BROTLI
x[i++] = HTTP_ACCEPT_ENCODING_BR;
#endif
#ifdef USE_BZ2LIB
x[i++] = HTTP_ACCEPT_ENCODING_BZIP2
| HTTP_ACCEPT_ENCODING_X_BZIP2;
#endif
#ifdef USE_ZLIB
x[i++] = HTTP_ACCEPT_ENCODING_GZIP
| HTTP_ACCEPT_ENCODING_X_GZIP
| HTTP_ACCEPT_ENCODING_DEFLATE;
#endif
x[i] = 0; /* end of list */
return x;
}
}
SETDEFAULTS_FUNC(mod_deflate_set_defaults) {
static const config_plugin_keys_t cpk[] = {
{ CONST_STR_LEN("deflate.mimetypes"),
T_CONFIG_ARRAY_VLIST,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.allowed-encodings"),
T_CONFIG_ARRAY_VLIST,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.max-compress-size"),
T_CONFIG_INT,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.min-compress-size"),
T_CONFIG_SHORT,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.compression-level"),
T_CONFIG_SHORT,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.output-buffer-size"),
T_CONFIG_SHORT,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.work-block-size"),
T_CONFIG_SHORT,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.max-loadavg"),
T_CONFIG_STRING,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.cache-dir"),
T_CONFIG_STRING,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("compress.filetype"),
T_CONFIG_ARRAY_VLIST,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("compress.allowed-encodings"),
T_CONFIG_ARRAY_VLIST,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("compress.cache-dir"),
T_CONFIG_STRING,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("compress.max-filesize"),
T_CONFIG_INT,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("compress.max-loadavg"),
T_CONFIG_STRING,
T_CONFIG_SCOPE_CONNECTION }
,{ CONST_STR_LEN("deflate.params"),
T_CONFIG_ARRAY_KVANY,
T_CONFIG_SCOPE_CONNECTION }
,{ NULL, 0,
T_CONFIG_UNSET,
T_CONFIG_SCOPE_UNSET }
};
plugin_data * const p = p_d;
if (!config_plugin_values_init(srv, p, cpk, "mod_deflate"))
return HANDLER_ERROR;
/* process and validate config directives
* (init i to 0 if global context; to 1 to skip empty global context) */
for (int i = !p->cvlist[0].v.u2[1]; i < p->nconfig; ++i) {
config_plugin_value_t *cpv = p->cvlist + p->cvlist[i].v.u2[0];
for (; -1 != cpv->k_id; ++cpv) {
switch (cpv->k_id) {
case 9: /* compress.filetype */
log_error(srv->errh, __FILE__, __LINE__,
"DEPRECATED: %s replaced with deflate.mimetypes",
cpk[cpv->k_id].k);
cpv->k_id = 0; /* deflate.mimetypes */
__attribute_fallthrough__
case 0: /* deflate.mimetypes */
/* mod_deflate matches mimetype as prefix of Content-Type
* so ignore '*' at end of mimetype for end-user flexibility
* in specifying trailing wildcard to grouping of mimetypes */
for (uint32_t m = 0; m < cpv->v.a->used; ++m) {
buffer *mimetype=&((data_string *)cpv->v.a->data[m])->value;
[multiple] reduce redundant NULL buffer checks This commit is a large set of code changes and results in removal of hundreds, perhaps thousands, of CPU instructions, a portion of which are on hot code paths. Most (buffer *) used by lighttpd are not NULL, especially since buffers were inlined into numerous larger structs such as request_st and chunk. In the small number of instances where that is not the case, a NULL check is often performed earlier in a function where that buffer is later used with a buffer_* func. In the handful of cases that remained, a NULL check was added, e.g. with r->http_host and r->conf.server_tag. - check for empty strings at config time and set value to NULL if blank string will be ignored at runtime; at runtime, simple pointer check for NULL can be used to check for a value that has been set and is not blank ("") - use buffer_is_blank() instead of buffer_string_is_empty(), and use buffer_is_unset() instead of buffer_is_empty(), where buffer is known not to be NULL so that NULL check can be skipped - use buffer_clen() instead of buffer_string_length() when buffer is known not to be NULL (to avoid NULL check at runtime) - use buffer_truncate() instead of buffer_string_set_length() to truncate string, and use buffer_extend() to extend Examples where buffer known not to be NULL: - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL (though we might set it to NULL if buffer_is_blank(cpv->v.b)) - address of buffer is arg (&foo) (compiler optimizer detects this in most, but not all, cases) - buffer is checked for NULL earlier in func - buffer is accessed in same scope without a NULL check (e.g. b->ptr) internal behavior change: callers must not pass a NULL buffer to some funcs. - buffer_init_buffer() requires non-null args - buffer_copy_buffer() requires non-null args - buffer_append_string_buffer() requires non-null args - buffer_string_space() requires non-null arg
12 months ago
size_t len = buffer_clen(mimetype);
if (len > 2 && mimetype->ptr[len-1] == '*')
[multiple] reduce redundant NULL buffer checks This commit is a large set of code changes and results in removal of hundreds, perhaps thousands, of CPU instructions, a portion of which are on hot code paths. Most (buffer *) used by lighttpd are not NULL, especially since buffers were inlined into numerous larger structs such as request_st and chunk. In the small number of instances where that is not the case, a NULL check is often performed earlier in a function where that buffer is later used with a buffer_* func. In the handful of cases that remained, a NULL check was added, e.g. with r->http_host and r->conf.server_tag. - check for empty strings at config time and set value to NULL if blank string will be ignored at runtime; at runtime, simple pointer check for NULL can be used to check for a value that has been set and is not blank ("") - use buffer_is_blank() instead of buffer_string_is_empty(), and use buffer_is_unset() instead of buffer_is_empty(), where buffer is known not to be NULL so that NULL check can be skipped - use buffer_clen() instead of buffer_string_length() when buffer is known not to be NULL (to avoid NULL check at runtime) - use buffer_truncate() instead of buffer_string_set_length() to truncate string, and use buffer_extend() to extend Examples where buffer known not to be NULL: - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL (though we might set it to NULL if buffer_is_blank(cpv->v.b)) - address of buffer is arg (&foo) (compiler optimizer detects this in most, but not all, cases) - buffer is checked for NULL earlier in func - buffer is accessed in same scope without a NULL check (e.g. b->ptr) internal behavior change: callers must not pass a NULL buffer to some funcs. - buffer_init_buffer() requires non-null args - buffer_copy_buffer() requires non-null args - buffer_append_string_buffer() requires non-null args - buffer_string_space() requires non-null arg
12 months ago
buffer_truncate(mimetype, len-1);
}
if (0 == cpv->v.a->used) cpv->v.a = NULL;
break;
case 10:/* compress.allowed-encodings */
log_error(srv->errh, __FILE__, __LINE__,