Packages

tiktoken_ex

0.2.0

0.2.0 0.1.0

Pure-Elixir TikToken-style byte-level BPE tokenizer, with helpers for Kimi K2 encodings.

HexDocs

Current section

Files

Jump to

Readme 2 Versions 1 Dependency Dependants Files Activity

tiktoken_ex CHANGELOG.md

Raw

CHANGELOG.md

# Changelog

## 0.2.0 - 2025-12-27
- Added HuggingFace file resolution with local caching and injectable fetchers.
- Added `Kimi.from_hf_repo/2` with optional ETS encoding caching.
- Added `TiktokenEx.Cache` helper for opt-in encoding reuse.

## 0.1.0 - 2025-12-12
- Initial release.

Search filters

Type any of these into the search box. They can be combined with free text which searches package names and descriptions.

Operator	Description	Example
name:	Match package name. Supports * wildcards and repo/package form	name:phx* or name:hexpm/phoenix
description:	Full-text search of package descriptions	description:auth
depends:	Packages depending on a given package. Supports repo:package form	depends:ecto or depends:hexpm:ecto
build_tool:	Filter by build tool	build_tool:mix
updated_after:	Packages updated after an ISO8601 datetime	updated_after:2025-01-01T00:00:00Z
extra:	Match custom metadata (key,value). Nested keys are separated by commas	extra:license,MIT

Wrap values in double quotes to include spaces, e.g. name:"my package".