rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
- Version0.0.6
- R versionunknown
- LicenseMIT
- Needs compilation?Yes
- Last release11/06/2024
Documentation
Team
David Zimmermann-Kollenda
Roger Zurawicki
Show author detailsRolesAuthorAuthors of the dependent Rust crates
Show author detailsRolesAuthor
Insights
Last 30 days
The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.
Last 365 days
The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.
Data provided by CRAN
Binaries
Dependencies
- Suggests1 package