Add PackTab for `CompositeProps` by taj-p · Pull Request #569 · linebender/parley

taj-p · 2026-03-03T21:03:57Z

Intent

Adds https://github.com/behdad/packtab.rs for generating CompositeProps behind a feature flag.

Results

I'm seeing roughly 115 kB savings on binary size (building vello_cpu_render with and without PackTab):

Overall layout performance seems a touch faster:

Tested via:

cargo export target/benchmarks -- bench --bench=main
cargo bench -q --bench=main --features parley_bench/packtab -- compare target/benchmarks/main -p --time 5

These results seem unsurprising considering taj-p#4.

Happy to remove the feature flagging and release to main directly. Also happy to keep it defensively behind a flag for a few weeks.

cc @behdad

behdad · 2026-03-04T16:39:12Z

Is there some of those attributes that packtab itself should generate? We already generate some.

Also, is a trailing newline missing? I can fix.

nicoburns

Not sure if this is waiting on approval. Consider this a rubber stamp approval (+ a review of the "plumbing"). I can't comment on whether the new data is actually valid.

nicoburns · 2026-03-05T12:28:48Z

parley_data/src/generated_packtab/mod.rs

+    2, 114, 2, 2, 2, 2, 2, 2, 2, 2, 115, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 116, 2, 2, 2, 2, 2, 2, 2, 2,
+    2, 2, 2, 2, 2, 117, 2, 118, 0, 0, 0, 0, 2, 119, 0, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 120, 2, 2, 2,
+    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 121, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,


This data looks very repetitive, which seems weird for compressed data...

PackTab just converts the data to a branchless multi-level lookup table. In this case, the shape of the table (from the comment in the generated code) is: [2^8,2^5,2^3,2^1]. That how it is broken down. You still get lots of repetition because eg. all different blocks of 32 numbers are encoded separately.

Interesting. So presumably it would be possible to further decrease the binary size impact by running the packtab'd data through a general purpose compression algorithm (LZ4, gzip, etc), at the cost of slightly higher RAM usage and a small one-time runtime cost to decompress the data.

If that kind of minimization is desired, I'm also curious to see what compression=10 does.

If my understanding is correct then the cost of compression=10 would be paid for every lookup? Whereas if the was compressed, it could be uncompressed into the compression=5 (or whatever) format at a one-time cost?

You are correct. I just became curious.

If the primary concern is woff serving, isn't that handled by compression in the transport layer?

If that kind of minimization is desired, I'm also curious to see what compression=10 does.

Performance seems a bit worse and the binary decreased in size by only 128 bytes.

If my understanding is correct then the cost of compression=10 would be paid for every lookup? Whereas if the was compressed, it could be uncompressed into the compression=5 (or whatever) format at a one-time cost?

Great idea! I'll create an issue after this is merged documenting it.

nicoburns · 2026-03-05T12:29:43Z

.github/workflows/ci.yml

        run: cargo run --locked -p parley_data_gen -- ./parley_data/src/generated

+      - name: regenerate unicode data (PackTab)
+        run: cargo run --locked -p parley_data_gen -- ./parley_data/src/generated_packtab --packtab --compression=5


Why level 5? Why not level 9? Is it because higher compression levels also affect decompression speed?

Correct.

In this case, it's unlikely that level 9 generate any different result. There's level 10 that gives you absolute smallest data size with whatever speed comes with it. The level numbers from 1 to 9 tune the heuristic to pick which solution in the tradeoff space (number of lookups vs data size) to pick.

it's unlikely that level 9 generate any different result

Behdad is right! Levels 5 through 9 produced the same result

nicoburns · 2026-03-05T12:31:23Z

parley_data_gen/src/lib.rs

+    code.push_str(&format!(
+        "#[inline]\npub fn composite_get(cp: u32) -> u32 {{\n    {namespace}_get(cp as usize)\n}}\n"
+    ));


Nit: you should be able to write!(code, ...) (or writeln) directlty into code.

Thank you! Updated in 6d5e2a8

taj-p · 2026-03-10T20:04:08Z

Is there some of those attributes that packtab itself should generate? We already generate some.

Hey @behdad - I noticed that packtab generates these fields into separate tables. Our parley_data_gen creates a single table of packed u32 that contains all the properties we care about so that we only perform 1 lookup on this table during analysis.

I'm curious what your thoughts on this approach are. Is it better to split the data into separate tables (that are presumably better compressed?) but suffer additional lookups or is it better to do something as we've done here?

Happy to benchmark both approaches in time.

behdad · 2026-03-10T20:29:30Z

Is there some of those attributes that packtab itself should generate? We already generate some.

Oh I meant the rust attributes. I think you call them lints:

for lint in [
        "unsafe_code",
        "trivial_numeric_casts",
        "missing_docs",
        "clippy::allow_attributes_without_reason",
        "clippy::unseparated_literal_suffix",
        "clippy::double_parens",
        "clippy::unnecessary_cast",
    ]

As for separate tables or combined, I think if you can look up once, this approach is better in general. Can you tell me which properties you are packing?

taj-p · 2026-03-10T20:33:18Z

Is there some of those attributes that packtab itself should generate? We already generate some.

Oh I meant the rust attributes. I think you call them lints:
for lint in [
        "unsafe_code",
        "trivial_numeric_casts",
        "missing_docs",
        "clippy::allow_attributes_without_reason",
        "clippy::unseparated_literal_suffix",
        "clippy::double_parens",
        "clippy::unnecessary_cast",
    ] 

Ah, yes. I think it's probably better for PackTab to generate these.

As for separate tables or combined, I think if you can look up once, this approach is better in general. Can you tell me which properties you are packing?

script
gc
gib
bidi
is_emoji_or_pictograph (isEmoji || isExtendedPictograph)
is_variation_selector
is_region_indicator
is_mandatory_linebreak (LineBreak is MandatoryBreak, CarriageReturn, LineFeed, or NextLine)

I think we should also use PackTab for CodePointMap/Trie data here:

parley/parley/src/analysis/mod.rs

Lines 29 to 95 in 6d5e2a8

    
           impl AnalysisDataSources { 
        
               pub(crate) fn new() -> Self { 
        
                   Self 
        
               } 
        
               #[inline(always)] 
        
               pub(crate) fn properties(&self, c: char) -> Properties { 
        
                   Properties::get(c) 
        
               } 
        
               #[inline(always)] 
        
               pub(crate) fn grapheme_segmenter(&self) -> GraphemeClusterSegmenterBorrowed<'_> { 
        
                   const { GraphemeClusterSegmenter::new() } 
        
               } 
        
               #[inline(always)] 
        
               fn word_segmenter(&self) -> WordSegmenterBorrowed<'static> { 
        
                   const { WordSegmenter::new_for_non_complex_scripts(WordBreakInvariantOptions::default()) } 
        
               } 
        
               #[inline(always)] 
        
               fn line_segmenter(&self, word_break_strength: WordBreak) -> LineSegmenterBorrowed<'static> { 
        
                   match word_break_strength { 
        
                       WordBreak::Normal => { 
        
                           const { 
        
                               let mut opt = LineBreakOptions::default(); 
        
                               opt.word_option = Some(LineBreakWordOption::Normal); 
        
                               LineSegmenter::new_for_non_complex_scripts(opt) 
        
                           } 
        
                       } 
        
                       WordBreak::BreakAll => { 
        
                           const { 
        
                               let mut opt = LineBreakOptions::default(); 
        
                               opt.word_option = Some(LineBreakWordOption::BreakAll); 
        
                               LineSegmenter::new_for_non_complex_scripts(opt) 
        
                           } 
        
                       } 
        
                       WordBreak::KeepAll => { 
        
                           const { 
        
                               let mut opt = LineBreakOptions::default(); 
        
                               opt.word_option = Some(LineBreakWordOption::KeepAll); 
        
                               LineSegmenter::new_for_non_complex_scripts(opt) 
        
                           } 
        
                       } 
        
                   } 
        
               } 
        
               #[inline(always)] 
        
               fn composing_normalizer(&self) -> CanonicalCompositionBorrowed<'_> { 
        
                   const { CanonicalComposition::new() } 
        
               } 
        
               #[inline(always)] 
        
               fn decomposing_normalizer(&self) -> CanonicalDecompositionBorrowed<'_> { 
        
                   const { CanonicalDecomposition::new() } 
        
               } 
        
               #[inline(always)] 
        
               pub(crate) fn script_short_name(&self) -> PropertyNamesShortBorrowed<'static, Script> { 
        
                   PropertyNamesShort::new() 
        
               } 
        
               #[inline(always)] 
        
               fn brackets(&self) -> CodePointMapDataBorrowed<'_, BidiMirroringGlyph> { 
        
                   const { CodePointMapData::new() } 
        
               } 
        
           }

behdad · 2026-03-10T23:57:46Z

gib

Which is that?

Given what you have, if they all fit in a 32bit value, keep it that way. Some (variation selector, regional indicator) are a few operations to compute, but will probably won't save any bytes removing them from the composite.

behdad · 2026-03-11T15:20:52Z

for lint in [
        "unsafe_code",
        "trivial_numeric_casts",
        "missing_docs",
        "clippy::allow_attributes_without_reason",
        "clippy::unseparated_literal_suffix",
        "clippy::double_parens",
        "clippy::unnecessary_cast",
    ]

Ah, yes. I think it's probably better for PackTab to generate these.

How's this?
behdad/packtab.rs@126e0dd

I can make a release if it looks good.

Add PackTab behind feature flag

f402b44

taj-p changed the title ~~Add PackTab behind feature flag~~ Add PackTab for CompositeProps behind feature flag Mar 3, 2026

taj-p added 3 commits March 4, 2026 07:36

.

436fe91

.

f979c33

.

1f4c474

nicoburns approved these changes Mar 5, 2026

View reviewed changes

Use write

6d5e2a8

Use PackTab by default

39630e3

taj-p changed the title ~~Add PackTab for CompositeProps behind feature flag~~ Add PackTab for CompositeProps Mar 10, 2026

.

8d3be06

taj-p marked this pull request as ready for review March 10, 2026 20:37

.

10c0f8d

Conversation

taj-p commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

Results

Uh oh!

behdad commented Mar 4, 2026

Uh oh!

nicoburns left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taj-p commented Mar 10, 2026

Uh oh!

behdad commented Mar 10, 2026

Uh oh!

taj-p commented Mar 10, 2026

Uh oh!

behdad commented Mar 10, 2026

Uh oh!

behdad commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taj-p commented Mar 3, 2026 •

edited

Loading