oa::NnTransformer

Ready-to-train causal language model: token + position embeddings, a stack of Transformer blocks, final normalization, and vocabulary projection. Input token ids are [batch, contextLength]; logits are [batch*contextLength, vocabSize] for all-position next-token training.

ClC++ public headers

Inheritance

public Module

Public Methods

oa::I32 oa::NnTransformer::contextLength()

oa::Matrix oa::NnTransformer::forward(const oa::Matrix & inTokens)

oa::I32 oa::NnTransformer::hiddenWidth()

oa::I32 oa::NnTransformer::modelWidth()

oa::I32 oa::NnTransformer::numHeads()

oa::I32 oa::NnTransformer::numLayers()

oa::I32 oa::NnTransformer::vocabSize()

Examples

ExMl

Learn a 300-token BPE vocabulary, train a causal Transformer for 300 optimizer steps, verify that cross-entropy falls, and generate text from the trained model.

Constructor & Destructor Documentation

oa::NnTransformer::NnTransformer(
	oa::I32 inVocabSize,
	oa::I32 inContextLength,
	oa::I32 inModelWidth = 32,
	oa::I32 inHiddenWidth = 64,
	oa::I32 inNumLayers = 1,
	oa::I32 inNumHeads = 1,
	oa::F32 inEps = 1e-5F
)

No public source comment is attached to this declaration.

Parameters

inVocabSize

oa::I32

—

inContextLength

oa::I32

—

inModelWidth

oa::I32

Default: 32

inHiddenWidth

oa::I32

Default: 64

inNumLayers

oa::I32

Default: 1

inNumHeads

oa::I32

Default: 1

inEps

oa::F32

Default: 1e-5F

Public Method Documentation

oa::I32 oa::NnTransformer::contextLength()

No public source comment is attached to this declaration.

Returns

oa::I32

The declared return value.

oa::Matrix oa::NnTransformer::forward(
	const oa::Matrix & inTokens
)

No public source comment is attached to this declaration.

oa::I32

The declared return value.