rtiktoken

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

CRAN Package

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

  • Version0.0.6
  • R versionunknown
  • LicenseMIT
  • Needs compilation?Yes
  • Last release11/06/2024

Documentation


Team


Insights

Last 30 days

Last 365 days

The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.

Data provided by CRAN


Binaries


Dependencies

  • Suggests1 package