tokenizers

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

Last 30 days

This package has been downloaded 35,693 times in the last 30 days. The academic equivalent of having a dedicated subreddit. There are fans, and maybe even a few trolls! The following heatmap shows the distribution of downloads per day. Yesterday, it was downloaded 1,391 times.

The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.

Last 365 days

This package has been downloaded 352,802 times in the last 365 days. This is the kind of download count that makes grant committees nod approvingly. A job well done, even the stoic reviewers might be impressed! The day with the most downloads was Apr 30, 2024 with 1,746 downloads.

The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.