oa::AdamW

Decoupled weight decay — preferred for transformers.

ClC++ public headers

Inheritance

public Optimizer

Public Methods

oa::Status oa::AdamW::loadFrom(Engine & inEngine, const ModelFile & inFile)

oa::Status oa::AdamW::saveTo(Engine & inEngine, ModelFile & outFile)

void oa::AdamW::setLr(oa::F32 inLr)

void oa::AdamW::step()

oa::Status oa::AdamW::validateLoad(const ModelFile & inFile)

void oa::AdamW::zeroGrad()

Examples

ExMl

Learn a 300-token BPE vocabulary, train a causal Transformer for 300 optimizer steps, verify that cross-entropy falls, and generate text from the trained model.

Constructor & Destructor Documentation

oa::AdamW::AdamW(
	oa::Span<Parameter *> inParamPtrs,
	oa::F32 inLr = 1e-3f,
	oa::F32 inBeta1 = 0.9f,
	oa::F32 inBeta2 = 0.999f,
	oa::F32 inEps = 1e-8f,
	oa::F32 inWeightDecay = 0.01f
)

Composite modules: root params_ is often empty; pass pointers from subtree.

Parameters

inParamPtrs

oa::Span<Parameter *>

—

inLr

oa::F32

Default: 1e-3f

inBeta1

oa::F32

Default: 0.9f

inBeta2

oa::F32

Default: 0.999f

inEps

oa::F32

Default: 1e-8f

inWeightDecay

oa::F32

Default: 0.01f

oa::AdamW::AdamW(
	oa::Vector<Parameter *> & inParamPtrs,
	oa::F32 inLr = 1e-3f,
	oa::F32 inBeta1 = 0.9f,
	oa::F32 inBeta2 = 0.999f,
	oa::F32 inEps = 1e-8f,
	oa::F32 inWeightDecay = 0.01f
)

No public source comment is attached to this declaration.

Parameters

inParamPtrs

oa::Vector<Parameter *> &

—

inLr

oa::F32

Default: 1e-3f

inBeta1

oa::F32

Default: 0.9f

inBeta2

oa::F32

Default: 0.999f

inEps

oa::F32

Default: 1e-8f

inWeightDecay

oa::F32

Default: 0.01f

oa::AdamW::AdamW(
	oa::Vector<Parameter> & inParams,
	oa::F32 inLr = 1e-3f,
	oa::F32 inBeta1 = 0.9f,
	oa::F32 inBeta2 = 0.999f,
	oa::F32 inEps = 1e-8f,
	oa::F32 inWeightDecay = 0.01f
)

No public source comment is attached to this declaration.

Parameters

inParams

oa::Vector<Parameter> &

—

inLr

oa::F32

Default: 1e-3f

inBeta1

oa::F32

Default: 0.9f

inBeta2

oa::F32

Default: 0.999f

inEps

oa::F32

Default: 1e-8f

inWeightDecay

oa::F32

Default: 0.01f

Public Method Documentation

oa::Status oa::AdamW::loadFrom(
	Engine & inEngine,
	const ModelFile & inFile
)

No public source comment is attached to this declaration.

Parameters

inEngine

Engine &

—

inFile

const ModelFile &

—

Returns

oa::Status

The declared return value.

oa::Status oa::AdamW::saveTo(
	Engine & inEngine,
	ModelFile & outFile
)

No public source comment is attached to this declaration.

Parameters

inEngine

Engine &

—

outFile

ModelFile &

—

Returns

oa::Status

The declared return value.

void oa::AdamW::setLr(
	oa::F32 inLr
)

No public source comment is attached to this declaration.

Parameters

inLr

oa::F32

—

Returns

void

The declared return value.

void oa::AdamW::step()

No public source comment is attached to this declaration.

Returns

void

The declared return value.

oa::Status oa::AdamW::validateLoad(
	const ModelFile & inFile
)

No public source comment is attached to this declaration.

Parameters

inFile

const ModelFile &

—

Returns

oa::Status

The declared return value.

void oa::AdamW::zeroGrad()

No public source comment is attached to this declaration.

Returns

void

The declared return value.