Merge branch 'master' of https://github.com/ray-project/ray into eoak…

…es/metrics-caching
ray-project · Jan 21, 2025 · e9089a2 · e9089a2
2 parents 20b09b8 + f70dfed
commit e9089a2
Show file tree

Hide file tree

Showing 146 changed files with 2,425 additions and 1,488 deletions.
diff --git a/BUILD.bazel b/BUILD.bazel
@@ -589,6 +589,7 @@ ray_cc_library(
     ],
     deps = [
         "//src/ray/util",
+        "//src/ray/util:size_literals",
         "@com_github_jupp0r_prometheus_cpp//pull",
         "@com_google_absl//absl/base:core_headers",
         "@com_google_absl//absl/container:flat_hash_map",
@@ -786,6 +787,7 @@ ray_cc_library(
         ":worker_rpc",
         "//src/ray/protobuf:worker_cc_proto",
         "//src/ray/util",
+        "//src/ray/util:shared_lru",
         "@boost//:circular_buffer",
         "@boost//:fiber",
         "@com_google_absl//absl/cleanup",

diff --git a/doc/source/rllib/algorithm-config.rst b/doc/source/rllib/algorithm-config.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-algo-configuration-docs:
 
 AlgorithmConfig API
 ===================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 RLlib's :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` API is
 the auto-validated and type-safe gateway into configuring and building an RLlib
 :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`.

diff --git a/doc/source/rllib/checkpoints.rst b/doc/source/rllib/checkpoints.rst
@@ -1,14 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-checkpoints-docs:
 
 Checkpointing
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 RLlib offers a powerful checkpointing system for all its major classes, allowing you to save the
 states of :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instances and their subcomponents
 to local disk or cloud storage, and restore previously run experiment states and individual subcomponents.

diff --git a/doc/source/rllib/external-envs.rst b/doc/source/rllib/external-envs.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-external-env-setups-doc:
 
-
 External Environments and Applications
 ======================================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 In many situations, it doesn't make sense for an RL environment to be "stepped" by RLlib.
 For example, if you train a policy inside a complex simulator that operates its own execution loop,
 like a game engine or a robotics simulation. A natural and user friendly approach is to flip this setup around

diff --git a/doc/source/rllib/hierarchical-envs.rst b/doc/source/rllib/hierarchical-envs.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-hierarchical-environments-doc:
 
-
 Hierarchical Environments
 =========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 You can implement hierarchical training as a special case of multi-agent RL. For example, consider a two-level hierarchy of policies,
 where a top-level policy issues high level tasks that are executed at a finer timescale by one or more low-level policies.
 The following timeline shows one step of the top-level policy, which corresponds to four low-level actions:

diff --git a/doc/source/rllib/key-concepts.rst b/doc/source/rllib/key-concepts.rst
@@ -1,14 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-key-concepts:
 
 Key concepts
 ============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 To help you get a high-level understanding of how the library works, on this page, you learn about the
 key concepts and general architecture of RLlib.
 

diff --git a/doc/source/rllib/multi-agent-envs.rst b/doc/source/rllib/multi-agent-envs.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-multi-agent-environments-doc:
 
 Multi-Agent Environments
 ========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 In a multi-agent environment, multiple "agents" act simultaneously, in a turn-based
 sequence, or through an arbitrary combination of both.
 

diff --git a/doc/source/rllib/new-api-stack-migration-guide.rst b/doc/source/rllib/new-api-stack-migration-guide.rst
@@ -1,8 +1,5 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-new-api-stack-migration-guide:
 
 .. testcode::
@@ -15,6 +12,8 @@
 New API stack migration guide
 =============================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 This page explains, step by step, how to convert and translate your existing old API stack
 RLlib classes and code to RLlib's new API stack.
 

diff --git a/doc/source/rllib/package_ref/algorithm-config.rst b/doc/source/rllib/package_ref/algorithm-config.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _algorithm-config-reference-docs:
 
-
 Algorithm Configuration API
 ===========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 .. currentmodule:: ray.rllib.algorithms.algorithm_config
 
 Constructor

diff --git a/doc/source/rllib/package_ref/algorithm.rst b/doc/source/rllib/package_ref/algorithm.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _algorithm-reference-docs:
 
 Algorithms
 ==========
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 The :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` class is the highest-level API in RLlib responsible for **WHEN** and **WHAT** of RL algorithms.
 Things like **WHEN** should we sample the algorithm, **WHEN** should we perform a neural network update, and so on.
 The **HOW** will be delegated to components such as ``RolloutWorker``, etc..
@@ -56,8 +56,8 @@ Algorithm API
 
 .. currentmodule:: ray.rllib.algorithms.algorithm
 
-Constructor
-~~~~~~~~~~~
+Construction and setup
+~~~~~~~~~~~~~~~~~~~~~~
 
 .. autosummary::
     :nosignatures:
@@ -66,50 +66,46 @@ Constructor
     ~Algorithm
     ~Algorithm.setup
     ~Algorithm.get_default_config
+    ~Algorithm.env_runner
+    ~Algorithm.eval_env_runner
+
 
-Inference and Evaluation
-~~~~~~~~~~~~~~~~~~~~~~~~
+Training
+~~~~~~~~
 .. autosummary::
     :nosignatures:
     :toctree: doc/
 
-    ~Algorithm.compute_actions
-    ~Algorithm.compute_single_action
-    ~Algorithm.evaluate
+    ~Algorithm.train
+    ~Algorithm.training_step
 
-Saving and Restoring
+Saving and restoring
 ~~~~~~~~~~~~~~~~~~~~
 .. autosummary::
     :nosignatures:
     :toctree: doc/
 
+    ~Algorithm.save_to_path
+    ~Algorithm.restore_from_path
     ~Algorithm.from_checkpoint
-    ~Algorithm.from_state
-    ~Algorithm.get_weights
-    ~Algorithm.set_weights
-    ~Algorithm.export_model
-    ~Algorithm.export_policy_checkpoint
-    ~Algorithm.export_policy_model
-    ~Algorithm.restore
-    ~Algorithm.restore_workers
-    ~Algorithm.save
-    ~Algorithm.save_checkpoint
+    ~Algorithm.get_state
+    ~Algorithm.set_state
 
 
-Training
-~~~~~~~~
+Evaluation
+~~~~~~~~~~
 .. autosummary::
     :nosignatures:
     :toctree: doc/
 
-    ~Algorithm.train
-    ~Algorithm.training_step
+    ~Algorithm.evaluate
 
 Multi Agent
 ~~~~~~~~~~~
 .. autosummary::
     :nosignatures:
     :toctree: doc/
 
+    ~Algorithm.get_module
     ~Algorithm.add_policy
     ~Algorithm.remove_policy
diff --git a/doc/source/rllib/package_ref/callback.rst b/doc/source/rllib/package_ref/callback.rst
@@ -1,10 +1,12 @@
-.. include:: /_includes/rllib/new_api_stack.rst
+.. include:: /_includes/rllib/we_are_hiring.rst
 
 .. _rllib-callback-reference-docs:
 
 Callback APIs
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 Callback APIs enable you to inject code into an experiment, an Algorithm,
 and the subcomponents of an Algorithm.
 

diff --git a/doc/source/rllib/package_ref/distributions.rst b/doc/source/rllib/package_ref/distributions.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-distributions-reference-docs:
 
 Distribution API
 ================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 .. currentmodule:: ray.rllib.models.distributions
 
 Base Distribution class

diff --git a/doc/source/rllib/package_ref/env.rst b/doc/source/rllib/package_ref/env.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _env-reference-docs:
 
 Environments
 ============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 RLlib mainly supports the `Farama gymnasium API <https://gymnasium.farama.org/>`__ for
 single-agent environments, and RLlib's own :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv`
 API for multi-agent setups.

diff --git a/doc/source/rllib/package_ref/env/env_runner.rst b/doc/source/rllib/package_ref/env/env_runner.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _env-runner-reference-docs:
 
 EnvRunner API
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.env_runner.EnvRunner
 ------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/multi_agent_env.rst b/doc/source/rllib/package_ref/env/multi_agent_env.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _multi-agent-env-reference-docs:
 
 MultiAgentEnv API
 =================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.multi_agent_env.MultiAgentEnv
 ---------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/multi_agent_env_runner.rst b/doc/source/rllib/package_ref/env/multi_agent_env_runner.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _multi-agent-env-runner-reference-docs:
 
 MultiAgentEnvRunner API
 =======================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.multi_agent_env_runner.MultiAgentEnvRunner
 ----------------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/multi_agent_episode.rst b/doc/source/rllib/package_ref/env/multi_agent_episode.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _multi-agent-episode-reference-docs:
 
 MultiAgentEpisode API
 =====================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.multi_agent_episode.MultiAgentEpisode
 -----------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/single_agent_env_runner.rst b/doc/source/rllib/package_ref/env/single_agent_env_runner.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _single-agent-env-runner-reference-docs:
 
 SingleAgentEnvRunner API
 ========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.single_agent_env_runner.SingleAgentEnvRunner
 ------------------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/single_agent_episode.rst b/doc/source/rllib/package_ref/env/single_agent_episode.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _single-agent-episode-reference-docs:
 
 SingleAgentEpisode API
 ======================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.single_agent_episode.SingleAgentEpisode
 -------------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/utils.rst b/doc/source/rllib/package_ref/env/utils.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _env-utils-reference-docs:
 
 Env Utils
 =========
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.utils
 ---------------