parity

Generate parity data (PAR2) from directories (resumable and linkable)

Usage

parity src

Notes

• Check Makefile for installation
• Check SCRIPT_PATH=parity path for paths
• Supports: unlimited number of files (generates multiple parity files)
• Tested on: 147 million files with long unicode paths
• Default settings produces single parity file if plausible, or a parity grouping directory if not
• Source file paths inside parity files are relative to provided argument path
• Parity files named from hashing source file metadata (allows incremental updating)
• Using parity files: find ${parity} -type f -exec par2 {verify|repair} -B ${source} '{}' ';'

Suggestions

Single parity

Produces single group attempting to group everything into one parity file, errors out if not plausible

• PARITY_FILE_LIMIT=1T

Updatable directory

Produces many groupings attempting to minimize large costly parity files and updates across different parity files

• PARITY_PER_DIR=yes
• PARITY_TIME_SORT=yes
• PARITY_SIZE_LIMIT=1G
• PARITY_FILE_LIMIT=128
• PARITY_ISOLATE=no

Many parity files

Produces groupings of parity files attempting to minimize directory breadth and skips finishing clean-up scan

• PARITY_CHECKSUM_SCALE=2
• PARITY_CLEAN=no

Archival parity

Produces groupings attempting to produce minimally sized parity files

• PARITY_SIZE_SCALE=2
• PARITY_SIZE_OFFSET=4

Verification parity

Produces parity files that are only capable of detecting errors, not correcting them

• PARITY_RECOVERY=0

Per file parity

Produce a parity file per usable source file

• PARITY_FILE_LIMIT=1

Options

Environment variables can be defined to modify behaviour

• script library: Check documentation for options
• PARITY_PER_DIR: generate a parity file per directory
• PARITY_CHECKSUM_SCALE: group parity files into n-length hash prefix directories (negative for suffix)
• PARITY_FILE_LIMIT: maximum number of source files per parity file
• PARITY_TIME_SORT: sort source files by modification time (before grouping)
• PARITY_TIME_SCALE: source file modification time resolution (when grouping)
• PARITY_SIZE_LIMIT: maximum size of source files per parity file (when grouped)
• PARITY_SIZE_SCALE: source file size resolution (when grouping)
• PARITY_SIZE_OFFSET: source file size at which to considering grouping (order of magnitude)
• PARITY_DST: destination path for parity files (extension omitted)
• PARITY_LINK: path to previous parity files (reused if plausible)
• PARITY_SLICE: size or number of source file slices (check parpar for documentation)
• PARITY_RECOVERY: desired recovery potential (check parpar for documentation)
• PARITY_STDIN: treat argument as base path and read file include list from STDIN (null delimited)
• PARITY_CLEAN: remove extraneous parity files (useful when updating)
• PARITY_ISOLATE: move single parity file to destination path and remove parity directory (if plausible)
• PARITY_COMPRESS: compress program for parity index file (must support -d for decompression)
• PARITY_CHECKSUM: hashing program for source file metadata and parity file naming
• PARITY_LINKER: linking program for parity file reusing (ln-like)

Dependencies

Executables must be available in PATH

• script library
• parpar
• numfmt
• zstd (if PARITY_COMPRESS is not defined)
• sha1sum (if PARITY_CHECKSUM is not defined)
• ln (if PARITY_LINKER is not defined)