oa::BpeTokenizer

Learns byte pair merges to build vocabulary from 256 base bytes to target size. Useful for compressing sequences and improving model efficiency.

ClC++ public headers

Public Types

Public Methods

oa::String oa::BpeTokenizer::decode(const oa::Vector<oa::I32> & inTokens)

oa::Vector<oa::I32> oa::BpeTokenizer::encode(const char * inText)

oa::Vector<oa::I32> oa::BpeTokenizer::encodePrompt(const char * inPrompt, oa::I32 inContextLen)

oa::Status oa::BpeTokenizer::load(const oa::String & inPath)

oa::I32 oa::BpeTokenizer::numMerges()

oa::Status oa::BpeTokenizer::save(const oa::String & inPath)

void oa::BpeTokenizer::train(const char * inText, oa::I32 inNumMerges)

oa::I32 oa::BpeTokenizer::vocabSize()

Examples

ExMl

Learn a 300-token BPE vocabulary, train a causal Transformer for 300 optimizer steps, verify that cross-entropy falls, and generate text from the trained model.

Constructor & Destructor Documentation

oa::BpeTokenizer::BpeTokenizer(
	oa::I32 inTargetVocab = 512
)

Create BPE tokenizer with target vocabulary size

Parameters

inTargetVocab

oa::I32

Default: 512

Public Method Documentation

oa::String oa::BpeTokenizer::decode(
	const oa::Vector<oa::I32> & inTokens
)

Decode BPE tokens back to text

Parameters

inTokens

const oa::Vector<oa::I32> &

—

Returns

oa::String

The declared return value.

oa::Vector<oa::I32> oa::BpeTokenizer::encode(
	const char * inText
)

Encode text to BPE tokens

Parameters

inText

const char *

—

Returns

oa::Vector<oa::I32>

The declared return value.

oa::Vector<oa::I32> oa::BpeTokenizer::encodePrompt(
	const char * inPrompt,
	oa::I32 inContextLen
)

Encode prompt with padding to context length

Parameters

inPrompt

const char *

—

inContextLen

oa::I32

—

Returns

oa::Vector<oa::I32>

The declared return value.

oa::Status oa::BpeTokenizer::load(
	const oa::String & inPath
)

No public source comment is attached to this declaration.

Parameters

inPath

const oa::String &

—

Returns

oa::Status

The declared return value.

oa::I32 oa::BpeTokenizer::numMerges()

get number of learned merges

Returns

oa::I32

The declared return value.

oa::Status oa::BpeTokenizer::save(
	const oa::String & inPath
)

Persist/load the learned merge ranks. The format is deterministic and architecture-independent so a training checkpoint can ship its text vocab.

Parameters

inPath

const oa::String &

—

Returns

oa::Status

The declared return value.

void oa::BpeTokenizer::train(
	const char * inText,
	oa::I32 inNumMerges
)

Train BPE merges on text corpus

Parameters

inText

const char *

—

inNumMerges

oa::I32

—

Returns

void

The declared return value.

oa::I32 oa::BpeTokenizer::vocabSize()

get current vocabulary size

Returns

oa::I32

The declared return value.