29 lines
32 KiB
Plaintext
29 lines
32 KiB
Plaintext
|
rust
|
|||
|
|
é~ #rustc 1.97.1 (8bab26f4f 2026-07-14)ÁõI*µ~&ævè¼JšÃ -b5d59729c8525f07Á ¬›IçTp�+õZòØu—Ó»Ø -085f534a869f02d5ÁŠ õ ZEROÁ is_xid_startÁ is_xid_continueÁ tablesÁ Align8Á
|
|||
|
|
À £ Align64Á
|
|||
|
|
À £ UNICODE_VERSIONÁ ASCII_STARTÁ ASCII_CONTINUEÁ CHUNKÁ
|
|||
|
|
TRIE_STARTÁ
|
|||
|
|
TRIE_CONTINUEÁ LEAFÁ T� � ” Ó — Ó üå5 ÓÓÓ d€ * _Á þÿÿþÿÿ !0x7fffffe07fffffe0000000000000000Á ÿþÿÿ‡þÿÿÁ !0x7fffffe87fffffe03ff000000000000Á @ 64Á › ± 411Á ± 1793Á € ± 7808ÁÜ÷ À ) û £ O
|
|||
|
|
!i:-Éð� b � i p ä¸ � & š £ ¿÷Yí}~1b� _ � f m
|
|||
|
|
õ º|Ý\ ât‚] Ð\’] ÷,Ÿ] °$¦] ˜l¬] ƒT»]
|
|||
|
|
ü Æd ü ¦ £ [![github]](https://github.com/dtolnay/unicode-ident) [![crates-io]](https://crates.io/crates/unicode-ident) [![docs-rs]](https://docs.rs/unicode-ident)Á §† ü«j g [github]: https://img.shields.io/badge/github-8da0cb?style=for-the-badge&labelColor=555555&logo=githubÁ ü–n k [crates-io]: https://img.shields.io/badge/crates.io-fc8d62?style=for-the-badge&labelColor=555555&logo=rustÁ ü…m j [docs-rs]: https://img.shields.io/badge/docs.rs-66c2a5?style=for-the-badge&labelColor=555555&logo=docs.rsÁ ó† D÷ <br>Á €† ü„N K Implementation of [Unicode Standard Annex #31][tr31] for determining whichÁ üÓ@ = `char` values are valid in programming language identifiers.Á ”† ü˜1 . [tr31]: https://www.unicode.org/reports/tr31/Á ʆ üÎN K This crate is a better optimized implementation of the older `unicode-xid`Á ü�L I crate. This crate uses less static storage, and is able to classify bothÁ üêP M ASCII and non-ASCII codepoints with better performance, 6× faster thanÁ ”» `unicode-xid`.Á Ά DÒÐ
|
|||
|
|
Û† üß ## Comparison of performanceÁ €† ü„J G The following table shows a comparison between five Unicode identifierÁ ¤Ï implementations.Á ä† üè$ ! - `unicode-ident` is this crate;Á ü� I F - [`unicode-xid`] is a widely used crate run by the "unicode-rs" org;Á ü× C @ - `ucd-trie` and `fst` are two data structures supported by theÁ ä›
|
|||
|
|
[`ucd-generate`] tool;Á ü¸
|
|||
|
|
= : - [`roaring`] is a Rust implementation of Roaring bitmap.Á ö
|
|||
|
|
† üú
|
|||
|
|
P M The *static storage* column shows the total size of `static` tables that theÁ üË = : crate bakes into your binary, measured in 1000s of bytes.Á ‰ † ü� J G The remaining columns show the **cost per call** to evaluate whether aÁ üØ G D single `char` has the XID\_Start or XID\_Continue Unicode property,Á ü
|
|||
|
|
M J comparing across different ratios of ASCII to non-ASCII codepoints in theÁ |î
|
|||
|
|
input data.Á þ
|
|||
|
|
† ü‚> ; [`unicode-xid`]: https://github.com/unicode-rs/unicode-xidÁ üÁ@ = [`ucd-generate`]: https://github.com/BurntSushi/ucd-generateÁ ü‚< 9 [`roaring`]: https://github.com/RoaringBitmap/roaring-rsÁ ¿† üÃA > | | static storage | 0% nonascii | 1% | 10% | 100% nonascii |Á ì… |---|---|---|---|---|---|Á ü£L I | **`unicode-ident`** | 10.3 K | 0.41 ns | 0.44 ns | 0.44 ns | 0.93 ns |Á üðJ G | **`unicode-xid`** | 12.0 K | 2.43 ns | 2.50 ns | 2.85 ns | 8.65 ns |Á ü»G D | **`ucd-trie`** | 10.4 K | 1.28 ns | 1.25 ns | 1.20 ns | 1.97 ns |Á üƒA > | **`fst`** | 144 K | 50.9 ns | 51.0 ns | 48.5 ns | 26.7 ns |Á üÅF C | **`roaring`** | 66.1 K | 4.28 ns | 4.22 ns | 4.25 ns | 4.61 ns |Á Œ† ü�N K Source code for the benchmark is provided in the *bench* directory of thisÁ üß: 7 repo and may be repeated by running `cargo criterion`.Á š† DžÐ
|
|||
|
|
§† ü«$ ! ## Comparison of data structuresÁ І ¤Ô #### unicode-xidÁ é† üíO L They use a sorted array of character ranges, and do a binary search to lookÁ ü½B ? up whether a given character lands inside one of those ranges.Á €† \„ ```rustÁ ü�" # const _: &str = stringify! {Á ü³6 3 static XID_Continue_table: [(char, char); 763] = [Á üê% " ('\u{30}', '\u{39}'), // 0-9Á ü�% " ('\u{41}', '\u{5a}'), // A-ZÁ <¶ # "Á \¾ …Á <ʘ% üÒ# ('\u{e0100}', '\u{e01ef}'),Á 4ö ];Á Dý # };Á <† ```Á Ž† ü’L I The static storage used by this data structure scales with the number ofÁ üßL I contiguous ranges of identifier codepoints in Unicode. Every table entryÁ ü¬L I consumes 8 bytes, because it consists of a pair of 32-bit `char` values.Á ù† üýI F In some ranges of the Unicode codepoint space, this is quite a sparseÁ üÇK H representation – there are some ranges where tens of thousands ofÁ ü“M J adjacent codepoints are all valid identifier characters. In other places,Á üáL I the representation is quite inefficient. A characater like `µ` (U+00B5)Á ü® L I which is surrounded by non-identifier codepoints consumes 64 bits in theÁ üû : 7 table, while it would be just 1 bit in a dense bitmap.Á ¶ † üº N K On a system with 64-byte cache lines, binary searching the table touches 7Á ü‰ F C cache lines on average. Each cache line fits only 8 table entries.Á üÐ N K Additionally, the branching performed during the binary search is probablyÁ üŸ 1 . mostly unpredictable to the branch predictor.Á Ñ † üÕ M J Overall, the crate ends up being about 6× slower on non-ASCII inputÁ ü£ " compared to the fastest crate.Á Æ † üÊ N K A potential improvement would be to pack the table entries more compactly.Á ü™!O L Rust's `char` type is a 21-bit integer padded to 32 bits, which means everyÁ üé!L I table entry is holding 22 bits of wasted space, adding up to 3.9 K. TheyÁ ü¶"M J could instead fit every table entry into 6 bytes, leaving out some of theÁ ü„#P M padding, for a 25% improvement in space used. With some cleverness it may beÁ üÕ#K H possible to fit in 5 bytes or even 4 bytes by storing a low char and anÁ ü¡$N K extent, instead of low char and high char. I don't expect that performanceÁ üð$P M would improve much but this could be the most efficient for space across allÁ üÁ%3 0 the libraries, needing only about 7 K to store.Á õ%† Œù% ####
|
|||
|
|
ÿÿÿôÎ\lò\ 33FFYYnnƒƒ|É]$Ð] Ó€ üž^% üß]> ; Whether the character has the Unicode property XID\_Start.Ád¥^ B
|
|||
|
|
chÁ²^üÂa( ü€aA > Whether the character has the Unicode property XID\_Continue.Á|Éa B
|
|||
|
|
|
|||
|
|
ÓwÙaT¹\¹
|
|||
|
|
skipÁlª\ „¨\ 4½\ 4‰ À �»� � ² O
|
|||
|
|
!i:-Éð �²BQ �²B
|
|||
|
|
�]_ d“ � <Ê À «z Ò ¿÷Yí}~1b �BH �B
|
|||
|
|
�]_ dÕ �üå' |ï Šüœ" \ Áüä% tõ Áì° ,Á ±üÕ/ Tç û £ O
|
|||
|
|
!i:-ÉðÓ± ›
|
|||
|
|
!%+/26:>¥C G K ïïïïïïïïïïïïPïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïORVZïïïïïïïïïïïïïïïïïïïïï0 î^bdhlpðtvz~‚†ŠŽ’–šžïA¢ åïï¤ïïïïïï)ï© Õ ï®Ë²ïïïïïïïïïïïïïï¶ ¬&¸ » ÄÆ ¹ÍÑØÜà ä ïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïçïïïïïïïïéïïïïïïïïïïëïïïïïïïïïïïïïïí ïÖ ïïïïïïïïïïïïïïïïïïïïïïïïïò ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿ ±üŸ3 l± û £ O
|
|||
|
|
!i:-ÉðÓ± ˆ #(-/48<@¥E I M ïïïïïïïïïïïïPïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïOTX\ïïïïïïïïïïïïïïïïïïïïï0 î`bfjnrðtx|€„ˆŒ�”˜œ ïA¢ åïï§ïïïïïï)ï© Ò ï°Â´ïïïïïïïïïïïïïï¶ ¬&¸ ¾ «¿¼ÁÄÈ Ê ¹ÏÔÚÞâ ä ª ïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïïçïïïïïïïïéïïïïïïïïïïëïïïïïïïïïïïïïïí ïÖ ïïïïïïïïïïïïïïïïïïïïïïïïïò ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿˆ$Ì ±üq+ $¿q š £ ¿÷Yí}~1bÓ± € €= ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÃÿ P ߸@×ÿÿûÿÿÿÿÿÿÿÿÿ¿ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÃÿ P ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ߸À×ÿÿûÿÿÿÿÿÿÿÿÿ¿ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿüÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿþÿÿÿÿÿÿÿÿ ÿÿÿ‡ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿûüÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿþÿÿÿÿÿÿÿÿþÿÿÿÿ¿¶ ÿÿÿ‡ ÿÿÿÿÿ Àþÿÿÿÿÿÿÿÿÿÿÿ/ `À œ ýÿÿÿ àÿÿÿÿÿÿÿÿÿÿ? üÿÿÿ0 ÿÿÿÿÿÿÿÿÿÿÃÿÿÿÿÿÿÿÿÿÿÿÿïŸÿýÿŸ ÿÿÿÿÿÿÿçÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ?$ÿÿ? ÿÿÿÿÿÿÿþ ÿÿÿÿÿ ðÿÿÿÿÿÿ# ÿ þÿáŸùÿÿýÅ# @ ° ÿÿÿÿÿ? ÿÿÿÿÿÿÿþ€ÿÿÿÿÿÿÿÿÿûÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÏÿþÿïŸùÿÿýÅóŸy€°ÏÿPà‡ùÿÿým ^ à¿ûÿÿýí# àŸùÿÿýí# ° èÇ=ÖÇÿ î‡ùÿÿýmÓ‡9^Àÿ? î¿ûÿÿýíó¿; Ïÿ þîŸùÿÿýíóŸ9à°Ïÿ ìÇ=ÖÇÿÃÇ=� Àÿ àßýÿÿýÿ# 7 áßýÿÿýï# p ðßýÿÿÿÿ' @p€ üàÿüÿÿû/ ÿßýÿÿýÿóß=`7Ïÿ ïßýÿÿýïóß=`pÏÿ ÿßýÿÿÿÿÿß}ð€Ïÿ üîÿüÿÿû/„_ÿÀÿ þÿÿÿÿÿ Ö÷ÿÿ¯ÿ _ ð ÿþÿÿÿ þÿÿÿÿÿÿÿÿ Ö÷ÿÿ¯ÿÿ?_ÿó ÿ Âÿþÿÿÿ þÿßÿÿþÿÿÿ @ ÿÿÿÿÿ € ?<bÀáÿ@ ÿÿÿÿ¿ ÿÿÿÿÿ÷ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ' ð ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ?ÿÿÿÿ¿ ÿÿÿÿÿ÷ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ==ÿÿÿÿÿ=ÿÿÿÿ==ÿÿÿÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿ??ÿÿÿÿÿÿÿÿÿ==ÿÿÿÿÿ=ÿÿÿÿ==ÿÿÿÿÿÿÿÿ=ÿÿÿÿÿÿÿÿç þ ÿÿ ÿÿÿÿÿÿÿÿÿÿ??þÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿøÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿŸÿÿþÿÿÿÿÿÿÿÿÿÿÿÇÿÿÿ€ÿÿ ÿÿ ÿß ÿÿÿÿÿÿ € ÿÿÿÿÿÿÿÿÿÿÿÿÿŸÿÿþÿÿÿÿÿÿÿÿÿÿÿÇÿÿÿ?€ÿÿ ÿÿ ÿß
|