Permutation trick

[aneubeck]

No description provided.

[Copilot]

Pull request overview

This PR adds a new consistent-choose-k crate providing stateless consistent hashing primitives (choose‑k ranking, per-key consistent permutations, and a deletion-tolerant node map), and updates benchmark/test dependencies across existing crates to newer versions.

Changes:

• Add the new crates/consistent-choose-k library with core algorithms, docs, examples, and benchmarks.
• Add the new benchmark workspace member and align Criterion usage (including switching benchmarks to std::hint::black_box).
• Bump dev/benchmark dependencies (e.g., criterion and tiktoken-rs / tokenizers) in existing crates.

Show a summary per file

File	Description
README.md	Lists the newly added consistent-choose-k crate in the repo overview.
Cargo.toml	Adds crates/consistent-choose-k/benchmarks to the workspace members.
crates/sparse-ngrams/Cargo.toml	Bumps Criterion dev-dependency version.
crates/sparse-ngrams/benchmarks/performance.rs	Updates benchmark black_box usage to std::hint::black_box.
crates/consistent-choose-k/Cargo.toml	Introduces the new crate’s package metadata and lib configuration.
crates/consistent-choose-k/src/lib.rs	Declares modules and exports the new crate’s public API surface.
crates/consistent-choose-k/src/choose_k.rs	Implements the consistent choose‑k ranking iterator and associated tests.
crates/consistent-choose-k/src/consistent_hash.rs	Implements the underlying consistent hash iterators and builder traits.
crates/consistent-choose-k/src/consistent_permutation.rs	Adds the “permutation trick” iterator (layered Feistel design) plus tests.
crates/consistent-choose-k/src/node_map.rs	Adds ConsistentNodeMap for deletion-tolerant assignment plus tests.
crates/consistent-choose-k/README.md	Adds crate-level documentation and proofs/derivations for the algorithm.
crates/consistent-choose-k/docs/permutation-design.md	Adds detailed design notes and rationale for the permutation construction.
crates/consistent-choose-k/examples/statistical_comparison.rs	Adds a chi-squared based statistical comparison example.
crates/consistent-choose-k/examples/bounded_load.rs	Adds a bounded-load consistent hashing example and comparisons.
crates/consistent-choose-k/benchmarks/Cargo.toml	Adds a dedicated benchmarks crate for consistent-choose-k.
crates/consistent-choose-k/benchmarks/performance.rs	Adds performance benchmarks comparing choose‑k vs permutation approaches.
crates/consistent-choose-k/benchmarks/criterion.toml	Adds Criterion chart/report configuration for the new benchmarks.
crates/bpe/Cargo.toml	Bumps tiktoken-rs dev-dependency version.
crates/bpe/tests/Cargo.toml	Bumps tiktoken-rs used by the test crate.
crates/bpe/benchmarks/Cargo.toml	Bumps tiktoken-rs and tokenizers versions in benchmarks.
crates/bpe-openai/Cargo.toml	Bumps tiktoken-rs dev-dependency version.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 21/21 changed files
• Comments generated: 4