Get the neural network weights out of a Tensorflow `Graph`

Question

I'm using RLlib to train a reinforcement learning policy (PPO algorithm). I want to see the weights in the neural network underlying the policy.

After digging through RLlib's PPO object, I found the TensorFlow Graph object. I thought that I would find the weights of the neural network there. But I can't find them. I see that this graph has ~1,000 nodes but I can't for the life of me find where TensorFlow is hiding the actual weights for the neural network. I looked through the nodes. I was told to keep an eye out for tf.Variable objects, but I couldn't find any. The closest thing I could find are nodes of type ReadVariableOp, but I couldn't find a tf.Variable in them. I did find a tf.Tensor in there, but I'm not sure whether it holds actual numbers, and if so how to get them.

Where do I find the weights of my neural network?

score 0 · Accepted Answer · edited Oct 22 '22 at 12:05

0

In a single-agent setup, do this:

weights = algo.get_policy().get_state()["weights"]

In a multi-agent setup, you'll need to specify the policy name:

weights = algo.get_policy(policy_name).get_state()["weights"]

edited Oct 22 '22 at 12:05

Ram Rachum

84,019
84
236
374

answered Oct 18 '22 at 11:56

Pedro Fillastre

892
6
10

`algo.get_policy()` just returns `None`, even after training. – Ram Rachum Oct 18 '22 at 15:21
how are you initializing your algo, `config = PPOConfig()` and `config.build` ? – Pedro Fillastre Oct 18 '22 at 15:35
Roughly like so: `from ray.rllib.algorithms.ppo import PPO; algorithm = PPO(config=config)` – Ram Rachum Oct 18 '22 at 19:33
I think you have a config issue, I update the code to launch a small config this will show you the weights – Pedro Fillastre Oct 19 '22 at 15:45
Your example helped me realize the difference. It should be `algorithm.get_policy('my_policy_name')` because I have a multi-agent setup. – Ram Rachum Oct 22 '22 at 12:04
Yes indeed if you have a multi-agent you need to get the policy of the desired agent, good to know you can pass the policy name – Pedro Fillastre Oct 22 '22 at 14:05

Get the neural network weights out of a Tensorflow `Graph`

1 Answers1