hub / github.com/dqbd/tiktokenizer

github.com/dqbd/tiktokenizer @main

Chat with this repo

repository ↗ · DeepWiki ↗ · + Follow

48 symbols 149 edges 29 files 0 documented · 0% 1 cross-repo links

What it actually does AI analysis from the code graph — generated when you open this

loading…

Extension points exported contracts — how you extend this code

Tokenizer (Interface)

(no doc) [3 implementers]

src/models/tokenizer.ts

ButtonProps (Interface)

(no doc)

src/components/Button.tsx

TokenizerResult (Interface)

(no doc)

src/models/tokenizer.ts

InputProps (Interface)

(no doc)

src/components/Input.tsx

TextAreaProps (Interface)

(no doc)

src/components/Input.tsx

Core symbols most depended-on inside this repo

src/sections/EncoderSelect.tsx

getTiktokenSegments

called by 3

src/utils/segments.ts

src/models/tokenizer.ts

encode

called by 2

src/pages/api/v1/encode.ts

src/models/tokenizer.ts

Shape

Function 31

Method 8

Interface 5

Class 4

Languages

TypeScript100%

Modules by API surface

src/models/tokenizer.ts15 symbols

src/pages/index.tsx4 symbols

src/server/api/trpc.ts3 symbols

src/sections/ChatGPTEditor.tsx3 symbols

src/models/index.ts3 symbols

src/utils/segments.ts2 symbols

src/utils/api.ts2 symbols

src/sections/TokenViewer.tsx2 symbols

src/sections/EncoderSelect.tsx2 symbols

src/pages/api/v1/encode.ts2 symbols

src/components/Input.tsx2 symbols

src/utils/cn.ts1 symbols

Used by 1 indexed graphs manifest dependencies, hub-wide

github.com/microsoft/vscode

For agents

$ claude mcp add tiktokenizer \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact