belief-dependent transition function

[gdaddi]

Hi,

Is there a way to implement a transition function that depends on the belief state? I checked the source code of both POMDPs and QuickPOMDP and there does not seem to be an obvious way of doing it. Thank you!
This is the pseudocode of what I'm trying to do:

    transition = function (s, a, b)
        if mean(particles(b)) < X:
            # transitionA
        else:
            # transitionB
        end
        # return new state 
    end,

[zsunberg]

Hi @gdaddi

There is not a way to do this because it would not conform to the definition of a POMDP that we are using. If you can describe the desired behavior, we may be able to find another way to implement it.

(also note that transition should return a distribution over states not a single new state as you have hinted at in the comment in your code.)

[gdaddi]

Hi @zsunberg

I am still trying to understand how to deal with belief in POMDP.jl.
In particular, now I am trying to an action that can reduce belief variance. The pseudocode of what I am trying to do is below.

if action = a
   belief = state_x
else
   propagate belief with transition 
end

Do you have any suggestion or examples of how to implement a similar structure?

Below there is some pseudocode of what I have tried. To represent and update belief I was using a BootstrapFilter filter, and I noted that the particles never coalesce to a unique position when action 0 is taken. Thank you!

    transition = function (s, a)
        if a == 0
            return Deterministic(state_x)
        else
            state_xp = f(s)
            return Deterministic(state_xp) 
        end      
    end,

    observation = function (a, sp)
        if a == 0
            return Deterministic(state_x)
        else
            covariance = f(sp)
            return Distribution(state_x, covariance)
        end
    end,

[zsunberg]

Hi @gdaddi ,

You are on the right track. In order to implement this, you will not need to change transition; you only need to change observation. The belief updater will use the information from observation to lower the belief entropy when the action is taken.

I would recommend starting with this simple light dark problem https://github.com/JuliaPOMDP/QuickPOMDPs.jl/blob/master/examples/lightdark.jl (it is similar to the one used here https://arxiv.org/abs/1709.06196), and modifying it to have an additional special action that reveals the state.

In that problem if you replace the observation with something like

observation = function (a, sp) 
    if a == :observe # whatever action you want to designate as observing
        return Normal(sp, 0.001) # low covariance. You could also use Deterministic here
    else
        return Normal(sp, abs(sp - light_loc) + 0.0001) # high covariance in the dark region at least
    end
end

and run it with a particle filter, I think you should get the desired behavior

Another example is subhunt. The agent can use the sonar to "ping" and get a more accurate observation with lower variance: https://github.com/zsunberg/SubHunt.jl/blob/master/src/observation.jl

[gdaddi]

Awesome, thank you! The observation function did the trick.

I am running into another issue that I have a hard time understanding. It seems like the script crashes when every particle in the belief state is terminal, but the true state is not.

Exception captured while planning; using default action 0   
exception = Planner failed to choose an action because all states sampled from the belief were terminal.

I have specified a default_action for this case solver = POMCPSolver(default_action=0), but it seems to be crashing regardless.

Do you have any ideas of what could be stopping the default action from being taken?

[zsunberg]

Hmmm... I think there must be another exception after the default action is taken. Can you post all of the error messages?

[gdaddi]

I've tried a few times to recreate that error after restarting my computer, and it seems to have disappeared. I will repost it in case it appears again. Thanks!