feat: add PositiveDefinite

[nicholaskl97]

A PositiveDefinite container wraps an underlying model and results in a model that returns a postive number whenever the input is nonzero (or not equal to a different point specified when defining the container). This is useful, among other applications, in neural Lyapunov applications.

[avik-pal]

Don't store a vector here. Instead pass in a initialization_function (ideally from WeightInitializers.jl) and construct the vector inside initialstates

[nicholaskl97]

I believe I've resolved this, but I'd like you to check if I did what you were asking when you get the chance.

[avik-pal]

same as above

[avik-pal]

pass in zeros32?

() -> copy(x0)

Here you can pass a dummy function that takes in (rng, in_dims) and ignores them

[nicholaskl97]

Do you mean something like the change I just made?

[avik-pal]

don't ignore the returned states here

[avik-pal]

same as above

[nicholaskl97]

If model is the mathematical function $$\phi$$, then ShiftTo is supposed to represent $$\phi(x) - \phi(x_0) + y_0$$, so that $$x_0$$ always gets mapped to $$y_0$$ (PositiveDefinite does something similar). In a sense, $$\phi(x)$$ and $$\phi(x_0)$$ are called at the same time in the mathematical representation. In my current code one has to be called first and using the output state of the first call as the input state of the second call means that my choice of which gets called first matters, but it really shouldn't. I figured it made the most sense just to use the state from the call $$\phi(x)$$.

Is it possible to combine the calls? I suppose I could call with hcat(x, st.in_val), but I don't know how I'd separate the last column back out without using the scalar indexing that CUDA hates so much.

[avik-pal]

import it as a package, not from Lux