Lines Matching +full:left +full:- +full:input +full:- +full:single +full:- +full:end
5 * This source code is licensed under both the BSD-style license (found in the
8 * You may select, at your option, one of the above-listed licenses.
36 real-time compression scenarios at zlib-level and better compression ratios.
37 The zstd compression library provides in-memory compression and decompression
41 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
47 - a single step (described as Simple API)
48 - a single step, reusing a context (described as Explicit context)
49 - unbounded multiple steps (described as Streaming compression)
53 - a single step (described as Simple dictionary API)
54 - a single step, reusing a dictionary (described as Bulk-processing
60 Advanced experimental APIs should never be used with a dynamically-linked
65 /*------ Version ------*/
95 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
109 * Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
130 * @return : - decompressed size of `src` frame content, if known
131 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
132 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too…
140 …* note 3 : decompressed size is always present when compression is completed using single-pass f…
142 * note 4 : decompressed size can be very large (64-bits value),
143 * potentially larger than what local system can handle as a single memory segment.
149 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
150 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
166 * or an error code if input is invalid */
171 …ize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
172 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
186 * and re-use it for each successive compression operation.
188 * Note : re-using context is just a speed / resource optimization.
190 * Note 2 : In multi-threaded environments,
213 * and re-use it for each successive compression operation.
239 * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
267 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
268 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
281 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
295 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
340 * default: windowlog - 7.
351 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
352 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
362 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
365 /* multi-threading parameters */
366 …/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_…
367 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
368 * In a situation where it's unknown if the linked library supports multi-threading or not,
373 …* ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back c…
378 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
391 … * - 0 means "default" : value will be determined by the library, depending on strategy
392 * - 1 means "no overlap"
393 * - 9 means "full overlap", using a full window size.
446 * - an error status field, which must be tested using ZSTD_isError()
447 * - lower and upper bounds, both inclusive
456 * Exception : when using multi-threading mode (nbWorkers >= 1),
465 * Total input data size to be compressed as a single frame.
467 * This value will also be controlled at end of frame, and trigger an error if not respected.
473 * It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
474 * Note 3 : Whenever all input data is provided and consumed in a single round,
489 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
495 * - The parameters : changes all parameters back to "default".
499 * - Both : similar to resetting the session, followed by resetting parameters.
507 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
508 * - The function is always blocking, returns when compression is completed.
535 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
560 * - an error status field, which must be tested using ZSTD_isError()
561 * - both lower and upper bounds, inclusive
588 const void* src; /*< start of input buffer */
589 size_t size; /*< size of input buffer */
601 /*-***********************************************************************
602 * Streaming compression - HowTo
607 * It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-us…
614 * it will re-use the same sticky parameters as previous compression session.
621 * consume input stream. The function will automatically update both `pos`
622 * fields within `input` and `output`.
623 * Note that the function may not consume the entire input, for example, because
624 * the output buffer is already full, in which case `input.pos < input.size`.
625 * The caller must check if input has been entirely consumed.
627 * and then present again remaining input data.
631 * consume some input, but if it can't it will wait for some, but not all,
637 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
638 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
683 * Behaves about the same as ZSTD_compressStream, with additional control on end directive.
684 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
685 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
686 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
687 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
688 * - endOp must be a valid directive
689 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
690 …* - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs …
693 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
694 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
696 …* if @return != 0, flush is not fully completed, there is still some data left within i…
699 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
706 ZSTD_inBuffer* input,
711 …* They are not required : ZSTD_compressStream*() happily accepts any buffer size, for both input a…
721 * for both input and output, to reduce the nb of roundtrips.
723 ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /*< recommended size for input buffer */
744 * Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).
746 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
747 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
749 …IB_API size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
756 /*-***************************************************************************
757 * Streaming decompression - HowTo
761 * ZSTD_DStream objects can be re-used multiple times.
764 * @return : recommended first input size
767 * Use ZSTD_decompressStream() repetitively to consume your input.
769 * If `input.pos < input.size`, some input has not been consumed.
773 * But if `output.pos == output.size`, there might be some data left within internal buffers.,
775 * Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZ…
779 * the return value is a suggested next input size (just a hint for b…
798 …_API size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
800 ZSTDLIB_API size_t ZSTD_DStreamInSize(void); /*!< recommended size for input buffer */
842 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
896 * It can still be loaded, but as a content-only dictionary. */
902 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
908 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
915 * - The frame does not require a dictionary to be decoded (most common case).
916 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
917 * Note : this use case also happens when using a non-conformant dictionary.
918 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
919 * - This is not a Zstandard frame.
930 * ZSTD_reset_session_and_parameters. Prefixes are single-use.
938 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
939 * meaning "return to no-dictionary mode".
941 * To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
943 * It's also a CPU consuming operation, with non-negligible impact on latency.
957 * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
958 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
961 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
968 * Reference a prefix (single-usage dictionary) for next compressed frame.
969 * A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
981 * It's a CPU consuming operation, with non-negligible impact on latency.
993 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
994 * meaning "return to no-dictionary mode".
996 * which has a non-negligible impact on CPU usage and latency.
1018 * Special: referencing a NULL DDict means "return to no-dictionary mode".
1024 * Reference a prefix (single-usage dictionary) to decompress next frame.
1027 * Prefix is **only used once**. Reference is discarded at end of frame.
1028 * End of frame is reached when ZSTD_decompressStream() returns 0.
1032 * Prefix buffer must remain unmodified up to the end of frame,
1076 * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
1102 #define ZSTD_FRAMEHEADERSIZE_PREFIX(format) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input s…
1118 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1135 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1146 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1155 /* --- Advanced types --- */
1181 * rep == 1 --> offset == repeat_offset_1
1182 * rep == 2 --> offset == repeat_offset_2
1183 * rep == 3 --> offset == repeat_offset_3
1185 * rep == 1 --> offset == repeat_offset_2
1186 * rep == 2 --> offset == repeat_offset_3
1187 * rep == 3 --> offset == repeat_offset_1 - 1
1208 …int checksumFlag; /*< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for…
1225 …ZSTD_dlm_byRef = 1 /*< Reference dictionary content -- the dictionary buffer must outlive its u…
1230 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1255 * - The contents of the CDict can be copied into the working context. This
1256 * means that the compression can search both the dictionary and input
1257 * while operating on a single set of internal tables. This makes
1258 * the compression faster per-byte of input. However, the initial copy of
1263 * - The CDict's tables can be used in-place. In this model, compression is
1264 * slower per input byte, because the compressor has to search two sets of
1265 * tables. However, this model incurs no start-up cost (as long as the
1269 * - The CDict's tables are not used at all, and instead we use the working
1273 * to the input size, and the input size is fairly large to begin with.
1301 ZSTD_ps_enable = 1, /* Force-enable the feature */
1313 * @return : - decompressed size of all data in all successive frames
1314 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1315 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1321 * note 3 : decompressed size can be very large (64-bits value),
1322 * potentially larger than what local system can handle as a single memory segment.
1327 …* note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input …
1336 * @return : - upper-bound for the decompressed size of all data in all successive frames
1337 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1340 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1342 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1343 * upper-bound = # blocks * min(128 KB, Window_Size)
1361 * Each block will end with a dummy sequence
1386 * @return : number of sequences left after merging
1393 * The entire source is compressed into a single frame.
1408 …* - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the ma…
1409 …* - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would …
1410 …* - ZSTD_c_windowLog affects offset validation: this function will return an error at higher de…
1413 …* Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep …
1426 * Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
1432 …* with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore i…
1443 * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
1472 * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
1473 * The estimate will assume the input may be arbitrarily large,
1484 * Note 2 : only single-threaded compression is supported.
1497 …Params() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compressi…
1498 * Note : CStream size estimation is only correct for single-threaded compression.
1521 * Initialize an object using a pre-allocated fixed-size buffer.
1523 * Provided pointer *must be 8-bytes aligned*.
1538 * Limitation 2 : static cctx currently not compatible with multi-threading.
1635 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1718 /* Force back-reference distances to remain < windowSize,
1735 * literals compression based on the compression parameters - specifically,
1773 * compression where the simulated compression of the dictionary left off.
1781 * structure for the dictionary that is read-optimized.
1801 * In general, you should expect compression to be faster--sometimes very much
1802 * so--and CDict creation to be slightly slower. Eventually, we will probably
1819 * When this flag is enabled zstd won't allocate an input window buffer,
1823 * avoid the memcpy() from the input buffer to the input window buffer.
1829 * this flag is ALWAYS memory safe, and will never access out-of-bounds
1845 * calls. Specifically: (out.size - out.pos) will never grow. This gives the
1853 * input window buffer (see ZSTD_c_stableInBuffer).
1855 * Zstd will check that (out.size - out.pos) never grows and return an error
1902 * Set to ZSTD_ps_disable to never use row-based matchfinder.
1903 * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
1906 * the row-based matchfinder based on support for SIMD instructions and the window log.
1918 * results than when the two buffers are non-contiguous. This flag forces zstd
1919 * to always load the prefix in non-contiguous mode, even if it happens to be
1941 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
1942 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
1946 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
1950 * - ZSTD_compressStream2() : Do compression using the CCtx.
1951 * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
1954 * for static allocation of CCtx for single-threaded compression.
2056 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
2071 * allowing selection between ZSTD_format_e input compression formats
2089 * an input buffer large enough to fit any compressed block. This will also
2091 * If you need to avoid the input buffer allocation use the buffer-less
2095 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2144 * This instruction is mandatory to decode data without a fully-formed header,
2208 * // Pseudocode: Set each zstd parameter and leave the rest as-is.
2240 * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
2267 * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
2270 …SrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end.
2281 unsigned long long ingested; /* nb input bytes read and buffered */
2282 unsigned long long consumed; /* nb input bytes actually compressed */
2290 * tells how much data has been ingested (read from input)
2291 * consumed (input actually compressed) and produced (output) for current frame.
2292 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
2342 * re-use decompression parameters from previous init; saves dictionary loading
2349 * Buffer-less and synchronous inner streaming functions
2357 Buffer-less streaming compression (synchronous mode)
2361 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
2367 Then, consume your input using ZSTD_compressContinue().
2369 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
2370 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
2371 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
2374 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
2376 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
2380 …It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
2383 `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
2386 /*===== Buffer-less streaming compression functions =====*/
2401 Buffer-less streaming decompression (synchronous mode)
2405 A ZSTD_DCtx object can be re-used multiple times.
2416 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
2426 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
2431 …which can @return an error code if required value is too large for current system (in 32-bits mode…
2433 …up to the moment there is not enough room left in the buffer to guarantee decoding another full bl…
2441 as long as the encoder and decoder progress in "lock-step",
2458 …Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInput…
2463 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
2466 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
2467 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
2468 c) Frame Content - any content (User Data) of length equal to Frame Size
2469 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
2473 /*===== Buffer-less streaming decompression functions =====*/
2490 …meHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); /*< doesn't consume input */
2518 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
2522 - Compressing and decompressing require a context structure
2524 - It is necessary to init context before starting
2528 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
2529 … + If input is larger than a block size, it's necessary to split input data into multiple blocks
2530 + For inputs larger than a single block, consider using regular ZSTD_compress() instead.
2532 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
2547 …kSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…