Lines Matching +full:cost +full:- +full:effective
5 * This source code is licensed under both the BSD-style license (found in the
8 * You may select, at your option, one of the above-listed licenses.
36 real-time compression scenarios at zlib-level and better compression ratios.
37 The zstd compression library provides in-memory compression and decompression
41 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
44 The lower the level, the faster the speed (at the cost of compression).
47 - a single step (described as Simple API)
48 - a single step, reusing a context (described as Explicit context)
49 - unbounded multiple steps (described as Streaming compression)
53 - a single step (described as Simple dictionary API)
54 - a single step, reusing a dictionary (described as Bulk-processing
60 Advanced experimental APIs should never be used with a dynamically-linked
65 /*------ Version ------*/
95 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
130 * @return : - decompressed size of `src` frame content, if known
131 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
132 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too…
140 …* note 3 : decompressed size is always present when compression is completed using single-pass f…
142 * note 4 : decompressed size can be very large (64-bits value),
149 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
150 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
171 …ize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
172 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
186 * and re-use it for each successive compression operation.
188 * Note : re-using context is just a speed / resource optimization.
190 * Note 2 : In multi-threaded environments,
213 * and re-use it for each successive compression operation.
239 * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
267 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
268 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
281 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
295 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
312 … * Note that currently, for all strategies < btopt, effective minimum is 4.
313 … * , for all strategies > fast, effective maximum is 6.
340 * default: windowlog - 7.
351 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
352 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
362 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
365 /* multi-threading parameters */
366 …/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_…
367 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
368 * In a situation where it's unknown if the linked library supports multi-threading or not,
378 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
391 … * - 0 means "default" : value will be determined by the library, depending on strategy
392 * - 1 means "no overlap"
393 * - 9 means "full overlap", using a full window size.
446 * - an error status field, which must be tested using ZSTD_isError()
447 * - lower and upper bounds, both inclusive
456 * Exception : when using multi-threading mode (nbWorkers >= 1),
489 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
495 * - The parameters : changes all parameters back to "default".
499 * - Both : similar to resetting the session, followed by resetting parameters.
507 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
508 * - The function is always blocking, returns when compression is completed.
535 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
560 * - an error status field, which must be tested using ZSTD_isError()
561 * - both lower and upper bounds, inclusive
601 /*-***********************************************************************
602 * Streaming compression - HowTo
607 * It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-us…
614 * it will re-use the same sticky parameters as previous compression session.
637 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
638 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
684 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
685 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
686 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
687 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
688 * - endOp must be a valid directive
689 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
690 …* - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs …
693 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
694 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
699 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
746 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
747 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
756 /*-***************************************************************************
757 * Streaming decompression - HowTo
761 * ZSTD_DStream objects can be re-used multiple times.
841 … The resulting state can be used for future compression operations with very limited startup cost.
842 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
896 * It can still be loaded, but as a content-only dictionary. */
902 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
908 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
915 * - The frame does not require a dictionary to be decoded (most common case).
916 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
917 * Note : this use case also happens when using a non-conformant dictionary.
918 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
919 * - This is not a Zstandard frame.
930 * ZSTD_reset_session_and_parameters. Prefixes are single-use.
938 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
939 * meaning "return to no-dictionary mode".
941 * To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
943 * It's also a CPU consuming operation, with non-negligible impact on latency.
957 * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
958 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
961 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
968 * Reference a prefix (single-usage dictionary) for next compressed frame.
981 * It's a CPU consuming operation, with non-negligible impact on latency.
993 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
994 * meaning "return to no-dictionary mode".
996 * which has a non-negligible impact on CPU usage and latency.
997 * It's recommended to "load once, use many times", to amortize the cost
1018 * Special: referencing a NULL DDict means "return to no-dictionary mode".
1024 * Reference a prefix (single-usage dictionary) to decompress next frame.
1036 * Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
1076 * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
1118 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1135 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1146 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1155 /* --- Advanced types --- */
1181 * rep == 1 --> offset == repeat_offset_1
1182 * rep == 2 --> offset == repeat_offset_2
1183 * rep == 3 --> offset == repeat_offset_3
1185 * rep == 1 --> offset == repeat_offset_2
1186 * rep == 2 --> offset == repeat_offset_3
1187 * rep == 3 --> offset == repeat_offset_1 - 1
1208 …int checksumFlag; /*< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for…
1225 …ZSTD_dlm_byRef = 1 /*< Reference dictionary content -- the dictionary buffer must outlive its u…
1230 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1255 * - The contents of the CDict can be copied into the working context. This
1258 * the compression faster per-byte of input. However, the initial copy of
1259 * the CDict's tables incurs a fixed cost at the beginning of the
1261 * the cost of the compression.
1263 * - The CDict's tables can be used in-place. In this model, compression is
1265 * tables. However, this model incurs no start-up cost (as long as the
1269 * - The CDict's tables are not used at all, and instead we use the working
1272 * This method is effective when the dictionary sizes are very small relative
1301 ZSTD_ps_enable = 1, /* Force-enable the feature */
1313 * @return : - decompressed size of all data in all successive frames
1314 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1315 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1321 * note 3 : decompressed size can be very large (64-bits value),
1336 * @return : - upper-bound for the decompressed size of all data in all successive frames
1337 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1340 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1342 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1343 * upper-bound = # blocks * min(128 KB, Window_Size)
1408 …* - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the ma…
1409 …* - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would …
1410 …* - ZSTD_c_windowLog affects offset validation: this function will return an error at higher de…
1413 …* Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep …
1426 * Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
1432 …* with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore i…
1443 * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
1472 * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
1484 * Note 2 : only single-threaded compression is supported.
1497 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1498 * Note : CStream size estimation is only correct for single-threaded compression.
1521 * Initialize an object using a pre-allocated fixed-size buffer.
1523 * Provided pointer *must be 8-bytes aligned*.
1538 * Limitation 2 : static cctx currently not compatible with multi-threading.
1635 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1718 /* Force back-reference distances to remain < windowSize,
1735 * literals compression based on the compression parameters - specifically,
1781 * structure for the dictionary that is read-optimized.
1801 * In general, you should expect compression to be faster--sometimes very much
1802 * so--and CDict creation to be slightly slower. Eventually, we will probably
1829 * this flag is ALWAYS memory safe, and will never access out-of-bounds
1845 * calls. Specifically: (out.size - out.pos) will never grow. This gives the
1855 * Zstd will check that (out.size - out.pos) never grows and return an error
1902 * Set to ZSTD_ps_disable to never use row-based matchfinder.
1903 * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
1906 * the row-based matchfinder based on support for SIMD instructions and the window log.
1918 * results than when the two buffers are non-contiguous. This flag forces zstd
1919 * to always load the prefix in non-contiguous mode, even if it happens to be
1927 * cost to memcpy() the data.
1941 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
1942 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
1946 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
1950 * - ZSTD_compressStream2() : Do compression using the CCtx.
1951 * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
1954 * for static allocation of CCtx for single-threaded compression.
2056 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
2091 * If you need to avoid the input buffer allocation use the buffer-less
2095 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2144 * This instruction is mandatory to decode data without a fully-formed header,
2208 * // Pseudocode: Set each zstd parameter and leave the rest as-is.
2240 * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
2267 * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
2292 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
2342 * re-use decompression parameters from previous init; saves dictionary loading
2349 * Buffer-less and synchronous inner streaming functions
2357 Buffer-less streaming compression (synchronous mode)
2361 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
2369 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
2370 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
2371 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
2374 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
2376 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
2383 `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
2386 /*===== Buffer-less streaming compression functions =====*/
2401 Buffer-less streaming decompression (synchronous mode)
2405 A ZSTD_DCtx object can be re-used multiple times.
2416 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
2426 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
2431 …which can @return an error code if required value is too large for current system (in 32-bits mode…
2441 as long as the encoder and decoder progress in "lock-step",
2463 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
2466 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
2467 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
2468 c) Frame Content - any content (User Data) of length equal to Frame Size
2469 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
2473 /*===== Buffer-less streaming decompression functions =====*/
2518 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
2522 - Compressing and decompressing require a context structure
2524 - It is necessary to init context before starting
2528 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
2532 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
2547 …kSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…