From ea7962d0b4916440e88e035e407830ecf861e263 Mon Sep 17 00:00:00 2001 From: Igor Gaponenko Date: Wed, 18 Dec 2024 11:31:48 -0800 Subject: [PATCH] Documentation on the http-based Worker Replication service --- doc/dev/api/TODO.rst | 34 + doc/dev/api/index.rst | 17 + doc/dev/api/introduction.rst | 22 + doc/dev/api/repl-worker.rst | 763 ++++++++++++++++++++++ doc/dev/index.rst | 1 + doc/ingest/api/reference/rest/general.rst | 2 + 6 files changed, 839 insertions(+) create mode 100644 doc/dev/api/TODO.rst create mode 100644 doc/dev/api/index.rst create mode 100644 doc/dev/api/introduction.rst create mode 100644 doc/dev/api/repl-worker.rst diff --git a/doc/dev/api/TODO.rst b/doc/dev/api/TODO.rst new file mode 100644 index 000000000..964421435 --- /dev/null +++ b/doc/dev/api/TODO.rst @@ -0,0 +1,34 @@ + +TODO +---- + + +Finish in a scope of the current ticket DM-42005 before the X-Mas break: + +- [**x**] Think about the locking mechanism of the method WorkerHttpRequest::toJson(). The method + acquires a lock on the mutext while the request may too have a lock on the same mutex + while processing the request in WorkerHttpRequest::execute(). This may result in a deadlock. + Perhaps no locking is needed as all since the resulting data are not lock sencitive? +- [**x**] Finish implementing a hierachy of the HTTP-based worker requests +- [**x**] Finish implementing the request processor for these requests +- [**x**] Add the new service to the Condfiguration and Registry to allow the Controller to send requests + to the worker via HTTP +- [**x**] Display connection parameters of the new service on the Web Dashboard +- [ ] Document the REST services in the documentation tree. +- [ ] Manually test the new implementation externally using ``curl`` or Python's ``requests`` module. + Think about the test cases to cover the new implementation. +- [ ] Extend the integration tests to cover the new implementation. + +Finish in a scope of a separate ticket during/after the X-Mas break: + +- [ ] Implement the MessengerHttp on the Controller side of the protocol. The class will + be providing the multiplexing API for the Controller to send requests to the worker. + The initial implementation will be based on the simple http::AsyncReq. +- [ ] Create a parallel hierarchy of the HTTP-based request & job classes on the Controller + side of the protocol. +- [ ] Test the new classes. +- [ ] Implement the MessengerHttp to reuse the socket connections for sending multiple requests + to the same worker. +- [ ] Test the new implementation to ensure it works the same way as the old one. +- [ ] Remove the old implementation of the Controller - Worker protocol. + diff --git a/doc/dev/api/index.rst b/doc/dev/api/index.rst new file mode 100644 index 000000000..49c4f4fda --- /dev/null +++ b/doc/dev/api/index.rst @@ -0,0 +1,17 @@ +.. note:: + + Information in this guide corresponds to the version **40** of the Qserv REST API. Keep in mind + that each implementation of the API has a specific version. The version number will change + if any changes to the implementation or the API that might affect users will be made. + The current document will be kept updated to reflect the latest version of the API. + +############################## +The internal REST API of Qserv +############################## + +.. toctree:: + :maxdepth: 4 + + introduction + repl-worker + TODO diff --git a/doc/dev/api/introduction.rst b/doc/dev/api/introduction.rst new file mode 100644 index 000000000..d2a3410b1 --- /dev/null +++ b/doc/dev/api/introduction.rst @@ -0,0 +1,22 @@ +.. _qserv-api-introduction: + +Introduction +============ + +The Qserv REST API is a collection of RESTful web services that provide access to various components of the Qserv system. +The API enforces a specific interaction model between the client and the server. The following highlights are worth mentioning: + +- All ``POST``, ``PUT`` and ``DELETE`` requests must be accompanied by a JSON payload. +- Responses of all but a few select services are in JSON format. Exceptions are documented in the API documentation. +- Schemas of the JSON requests and payloads are defined in the API documentation. +- The API is versioned. The version number is included in the URL path of the ``GET`` requests, and it's + included into the JSON payload of the ``POST``, ``PUT`` and ``DELETE`` requests. +- All API services are protected by an authentication mechanism. The client must provide a valid + authentication token in the JSON payload of the ``POST``, ``PUT`` and ``DELETE`` requests. + No authentication is required for the ``GET`` requests. + +The general information on the structure of the API can be found in the following document: + +- :ref:`ingest-general` + +The rest of the current document provides detailed information on the individual services that are available in the Qserv API. diff --git a/doc/dev/api/repl-worker.rst b/doc/dev/api/repl-worker.rst new file mode 100644 index 000000000..288e83960 --- /dev/null +++ b/doc/dev/api/repl-worker.rst @@ -0,0 +1,763 @@ +.. _qserv-api-repl-worker: + +Replication Worker Services +=========================== + +Scope +----- + +This document describes the Replication worker services in Qserv. The worker services are responsible for +processing requests that are submitted by the Replication Controller. The protocol is based on HTTP/JSON. + + +Categories +---------- + +There are two general categories of requests depending on the request processing mechanism. The first group includes +requests that are processed *asynchronously*. Once a request of this type is validated and accepted by the servivce, +it's put into a priority queue of the Replication worker in an order which depends on the *priority* level specific +in the request body. These requests also have an expiration timeout. If the request is not processed within the timeout, +it will be automatically cancelled by the server. Another importan attribute of the queued requests is their unique identifier. +The identifier is generated by the Controller and it's used to track the request status and to retrieve the results of +the request once it's finished. The following request types belong to this category: + +- :ref:`qserv-api-repl-worker-tests` (echo, etc.) +- :ref:`qserv-api-repl-worker-replica-management` (find, create, delete replicas) +- :ref:`qserv-api-repl-worker-sql` (SQL operations) + +The second group includes requests processed immediately by the service: + +- :ref:`qserv-api-repl-worker-request-management` (inspect and manage queued requests) +- :ref:`qserv-api-repl-worker-service-management` (inspect and manage the worker server) + +Request parameters +------------------ + +Even though the parameters of the requests are specific to the request type, there are many attributes that are common +to all requests. These attributes are sent with all request types as explained in the subsections below. + +Attributes sent with all request types +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following attributes are passed with any request type, regardless of the HTTP method: + +``instance_id`` : *string* + The mandatory identifier of a Qserv instance served by the Replication System. +``version`` : *number* = ``0`` + The optional version of the Qserv REST API. See :ref:`ingest-general-versioning` for more information on the API versioning. + +For the ``POST``, ``PUT`` and ``DELETE`` request types values of the attributes are send in the request body: + +.. code-block:: + + { + "instance_id" : , + "version" : + } + +For ``GET`` request types, parameters are specified in the URL query: + +.. code-block:: + + ?instance_id=&version= + +Authorization of the ``POST``, ``PUT`` and ``DELETE`` requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All requests that may modify the persistent state of Qserv or affect the state of the worker service must use the key-based +authentication mechanism: + +``auth_key`` : *string* + The required authentication key to access the service. The key must match the key set in the configuration of + the target worker service. + +This attribute's value is sent in the request body: + +.. code-block:: + + { + "auth_key" : + } + +.. _qserv-api-repl-worker-queued-request-params: + +JSON body of the queued requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All requests that are queued by the worker service before processed by the service are sent with the ``POST`` method. +The request body of each such request is a JSON object with the following **required** attributes: + +.. code-block:: + + { + "id" : , + "timeout" : , + "priority" : , + "req" : { + ... + } + } + +Where: + +``id`` : *string* + The unique ID of the request (generated by the Controller). +``timeout`` : *number* + The optional request expiration timeout. The timeout is meant for automatic cancelling/disposing requests + regardless of their statuses. The timeout is expressed in seconds since the *UNIX Epoch*. For requests where + the timeout is not set or where its value was set to ``0`` the worker-specific default value will be used. +``priority`` : *number* + The priority level of the request. Requests with higher priority levels are processed before the requests with + lower priority levels. +``req`` : *object* + The object containing the request-specific attributes. The schema of this object depends on the type of the request. + The payload of the object is documented in the relevant sections below. + + +Response objects +---------------- + +Responses returned by all worker services have the following attributes wich represent the common completion status +of the request: + +.. code-block:: + + { + "success" : , + "error" : , + "error_ext" : , + "warning" : , + + "status" : , + "status_str" : , + "status_ext" : , + "status_ext_str" : + } + +The first group represents the common attributes which are reported by all REST services in Qserv: + +``success`` : *number* + The completion status of the request. The value of ``1`` means that the request was successfully accepted + by the service after evaluating the input parameters and a context of the request. The value of ``0`` indicates + any problems with the operation. The error message will be provided in the ``error`` attribute. +``error`` : *string* + The error message in case of the failed request. +``error_ext`` : *object* + The extended error message in case of the failed request. +``warning`` : *string* + The optional warning message that may be posted in case of the successful request. + + +The second group of attributes represent the completion status codes which are specific to the worker services. These attributes +should be used only when ``success=1``: + +``status`` : *int* + The completion status of the operation. Values are defined in the C++ ``enum`` type ``protocol::Status``. +``status_str`` : *string* + The human readable representation of the above-defined completion status of the operation. +``status_ext`` : *int* + Extended status of this operation. Values are defined in the C++ ``enum`` type ``protocol::StatusExt::Status``. +``status_ext_str`` : *string* + The human readable representation of the above-defined completion status of the operation. + +Additional attributes returned by specific request types are documented in the relevant sections below. + +.. _qserv-api-repl-worker-response-queued: + +Responses of the queued requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + Responses of :ref:`qserv-api-repl-worker-request-management` requests adhere to the same schema as the queued requests. + +All responses have the following schema: + +.. code-block:: + + { + "id" : , + "priority" : , + "timeout" : , + "req" : , + + "type" : , + "expiration_timeout_sec" : , + + "performance" : { + "receive_time" : , + "start_time" : , + "finish_time" : + } + "result" : + } + + +The first group represents parameters of the original request (or a target request in case if the request management services +were called on the previously submitted queued requests): + +``id`` : *string* + The unique ID of the request (generated by the Controller). +``priority`` : *int* + The priority level of the request. +``timeout`` : *int* + The request expiration timeout that was passed in the original request (applies to the queued requests only). + The timeout is meant for automatic cancelling/disposing requests regardless of their statuses. The timeout is expressed + in seconds since the UNIX Epoch. For requests where the timeout is not set or where its value was set to ``0`` + the worker-specific default value will be used. The adjusted (effective) value of the timeout is reported in + the ``expiration_timeout_sec`` attribute. +``req`` : *object* + The original request object as it was received by the worker. + +The following attributes are assigned to a request by the worker server based on a nature of the request and its processing +context: + +``type`` : *string* + The type of the request. +``expiration_timeout_sec``: *int* + The effective expiration timeout of the request in seconds. + +The actual processing status of the request is reported in the following attributes: + +``performance`` : *object* + The current performance metrics of the request. Values of these parameters changes during request processing + before the request is finished. There are tree attributes in this object: + + - ``receive_time`` : *uint64_t* + When the request was received by a worker service (milliseconds since UNIX Epoch). A non-zero value + is guaranteeded for all requests that were received by the worker service. + + - ``start_time`` : *uint64_t* + When the request was started by a worker service (milliseconds since UNIX Epoch). A value of ``0`` + means that the request is still in the processing queue. + + - ``finish_time`` : *uint64_t* + When the request was finished by a worker service (milliseconds since UNIX Epoch). A value of ``0`` + means that the request is still in the processing queue or it's still being processed. + +``result`` : *object* + The result of the request. The schema of this object depends on the type of the request. Also note that + the payload of the ``result`` object depends on the type of the request: + + - The object is empty for all newely submitted requests that ended up in the processing queue and for requests which + are still being processed. + - The object is filled with the relevant data for the requests that is finished or failed. + + + +.. _qserv-api-repl-worker-response-service: + + +Responses of the service management requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TBC... + + + + +Request types +------------- + +.. _qserv-api-repl-worker-tests: + +Tests +^^^^^ + +.. note:: + + This group of request belongs the *queued* category. Requests of this type are processed by the worker service + asynchronously by a dedicated pool of the worker threads. Parameters of this request type are sent in the request + body as a JSON object. A schema of the request object is documented in :ref:`qserv-api-repl-worker-queued-request-params`. + A schema of the response object is documented in: :ref:`qserv-api-repl-worker-response-queued`. + +Echo +~~~~ + +The Controller sends a POST request to the Replication worker to test the functionality of the worker processor +and to simulate the request submittion/processing path. The ``echo`` request has no persistent side effects, such +as changes to the worker databases. Parameters of the request will be evaluated by the service. If all looks +okay then the request will be queued for processing. Otherwise, the service will return an error. + +.. code-block: + + POST /worker/echo + +These are the the request-specific attributes: + +.. code-block:: + + "req" : { + "delay" : , + "data" : + } + +Where: + +``delay`` : *int* + The delay in milliseconds before the response is sent back. +``data`` : *string* + The data to be echoed back. + +The schema of the ``result`` object in the responses pf the succesfully completed requests is presented below: + +.. code-block:: + + "results" : { + "data" : + } + +.. _qserv-api-repl-worker-replica-management: + +Replica management +^^^^^^^^^^^^^^^^^^ + + + +.. note:: + + This group of request belongs the *queued* category. Requests of this type are processed by the worker service + asynchronously by a dedicated pool of the worker threads. Parameters of this request type are sent in the request + body as a JSON object. A schema of the request object is documented in :ref:`qserv-api-repl-worker-queued-request-params`. + A schema of the response object is documented in: :ref:`qserv-api-repl-worker-response-queued`. + +TBC + +.. _qserv-api-repl-worker-sql: + +Database management +^^^^^^^^^^^^^^^^^^^ + +.. note:: + + This group of request belongs the *queued* category. Requests of this type are processed by the worker service + asynchronously by a dedicated pool of the worker threads. Parameters of this request type are sent in the request + body as a JSON object. A schema of the request object is documented in :ref:`qserv-api-repl-worker-queued-request-params`. + A schema of the response object is documented in: :ref:`qserv-api-repl-worker-response-queued`. + +TBC... + + +.. _qserv-api-repl-worker-request-management: + +Request management +^^^^^^^^^^^^^^^^^^ + +.. note:: + + This group of the **synchronous** requests are meant to monitor and manage the corresponding *queued* requests (the "target" requests). + Requests of this type are processed by the worker service instanteniously. Depending on the HTTP method, parameters of this request type + are sent either in in the request body as a JSON object or in the query string of the request URL. + A schema of the response object is documented in: :ref:`qserv-api-repl-worker-response-queued`. + +TBC... + +.. _qserv-api-repl-worker-service-management: + +Worker service management +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + This group of the **synchronous** requests are meant to monitor and manage the worker server itself. + Requests of this type are processed by the worker service instanteniously. Depending on the HTTP method, parameters of this request type + are sent either in in the request body as a JSON object or in the query string of the request URL. + A schema of the response object is documented in: :ref:`qserv-api-repl-worker-response-service`. + +TBC... + + + + + + + + + + + + + + + + +Replica management/information requests +--------------------------------------- + +All requests of this category are queued and processed by a dedicated pool of the worker threads. +Once the request is submitted and the worker service indicated that the request looked good, the state +of the request it can be further managed via: + +- TODO: link to the request tracking service +- TODO: link to the replica status service +- TODO: link to the replica cancel service + +Schemas of the response object for the successfully completed request varies depending on a type of a request. + +Schemas for the single replica requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The object has the following attributes: + +.. code-block:: + + { + "result" : { + "replica_info" : { + // enum ReplicaStatus { + // NOT_FOUND = 0; + // CORRUPT = 1; + // INCOMPLETE = 2; + // COMPLETE = 3; + // } + "status" : , // The status of the replica. Values corresponds to enums in class "ReplicaStatus" + "worker" : , // The worker ID + "database" : , + "chunk" : , + + // A collection of files + // + "file_info_many" : [ + + { + "name" : , // The name of a file + "size" : , // Size in bytes + "cs" : , // Control sum (if available) + "mtime" : , // The file content modification time in seconds (since UNIX Epoch) + + // The following parameters are set in the relevant contexts only. + // Otherwise they'll be set to some default value. + + "begin_transfer_time" : , // When the file migration started (where applies) [=0] + "end_transfer_time" : , // When the file migration finished (where applies) [=0] + "in_size" : // The size of an input file (where applies) [=0] + }, + ], + "verify_time" : // When the replica status was verified by a worker + } + } + } + +Schemas for the multi-replica requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + Presently, the only multi-replica request in tis category is the ``find-all`` request. + +The response object has the following attributes: + +.. code-block:: + + { + "result" : { + "replica_info_many" : [ + ... + ] + } + } + +Where each array entry is an object that has a single replica schema (``replica_info``) as described above for +the signle-replica requests. + + +Create a new chunk replica +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a POST request to the Replication worker to initiate the replica replica creation +operation on the target worker. Parameters of the request will be evaluated by the service. If all looks +okay then the request will be queued for processing. Otherwise, the service will return an error. + +.. code-block: + + POST /worker/replica/create + +The request-specific attributes: + +.. code-block:: + + { + "req" : { + "database" : , + "chunk" : , + "src_worker" : , // The source worker ID from where to pull the replica + "src_worker_host" : , // The source worker host (DNS or IP) + "src_worker_port" : // The source worker port + } + } + +Delete an existing chunk replica +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a POST request to the Replication worker to initiate the replica deletion +operation on the target worker. Parameters of the request will be evaluated by the service. If all looks +okay then the request will be queued for processing. Otherwise, the service will return an error. + +.. code-block: + + POST /worker/replica/delete + +The request-specific attributes: + +.. code-block:: + + { + "req" : { + "database" : , + "chunk" : + } + } + +Find info an existing chunk replica +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a POST request to the Replication worker to locate and report a status of a single chunk replica: + +.. code-block: + + POST /worker/replica/find + +The request-specific attributes: + +.. code-block:: + + { + "req" : { + "database" : , + "chunk" : , + "compute_cs" : // Compute the control sum of the replica files if not 0 + } + } + +Find info on all existing chunk replicas if a database +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a POST request to the Replication worker to locate and report a status of all chunk replicas +in a given database: + +.. code-block: + + POST /worker/replica/find-all + +The request-specific attributes: + +.. code-block:: + + { + "req" : { + "database" : + } + } + +Database management (SQL) Requests +---------------------------------- + + +Management requests +------------------- + +Tracking requests +^^^^^^^^^^^^^^^^^ + +The Controller sends a GET request to the Replication worker to track the status of the previously made +request and to retrieve results of the request if it's finished. The request URL should contain the unique +identifier ``id`` of the target request: + +.. code-block: + + GET /worker/request/track/:id + +In case of the successful request completion, the response object will not be empty and it will contain +the results of the request: + +.. code-block:: + + { + "result" : { + ... + } + } + +Retreiving request status +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a GET request to the Replication worker to get the status of the previously made +request. The request URL should contain the ID of the unique +identifier ``id`` of the target request: + +.. code-block: + + GET /worker/request/status/:id + +Note, that unlike the ``track`` request, the ``status`` request does not return the results of the request. +The result object will be present but it will be empty: + +.. code-block:: + + { + "result" : {} + } + +Stopping/cancelling requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a PUT request to the Replication worker to stop the previously made request: + +.. code-block: + + PUT /worker/request/stop/:id + +There are no request-specific attributes in the request object. + +Note, that unlike the ``track`` request, the ``stop`` request does not return the results of the request. +The result object will be present but it will be empty: + +.. code-block:: + + { + "result" : {} + } + +Disposing completed requests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There is a special request that's meant to be used by the Controller to dispose the completed +requests from the worker's internal storage. The request is sent as a POST request: + +.. code-block: + + POST /worker/request/dispose + +Where the request object is required to provide a collection (array) of the request IDs to be disposed: + +.. code-block:: + + { + "req" : { + "ids" : [ + , + ... + + ] + } + } + +The response object will have the completion status of the operation for each identifier mentioned in the request: + +.. code-block:: + + { + "result" : { + "ids_disposed" : { + : , + ... + : + } + } + } + +Where the value of the integer is the completion status of the operation. The value of ``1`` means that the request +was disposed successfully. The value of ``0`` means that the request was not found in a collection of the completed +requests + +Worker service management requests +---------------------------------- + +Requests in this category are meant to provide the Controller with the information on the worker service itself. +There are the following requests in this category: + +- TODO: link to: Get the worker status +- TODO: link to: Get info on requests at various stages of processing +- TODO: link to: Suspend the worker service +- TODO: link to: Resume the worker service +- TODO: link to: Drain requests at the worker service +- TODO: link to: Reconfigure the worker service + +The request-specific attributes are not required for these requests. + +Response objects of all service management requests have the following schema: + +.. code-block:: + + { + "status" : , // The completion status of the operation. Values corresponds to protocol::Status + "status_ext" : , // Extended status of this operation. Values corresponds to protocol::StatusExt [=NONE] + + "service_state" : , // The state of the worker service as defined in protocol::ServiceState + + "num_new_requests" : , + "num_in_progress_requests" : , + "num_finished_requests" : , + + "new_requests" : [ + ... + ], + "in_progress_requests" : [ + ... + ], + "finished_requests" : [ + ... + ] + } + +.. note:: + + The ``new_requests``, ``in_progress_requests``, and ``finished_requests`` are arrays of the request objects + that are in the corresponding state. These collections will not be empty only for the following request types: + + - Get info on requests at various stages of processing + - Drain requests at the worker service + + The schema of the request descriptors is the same as the schema of the corresponding original request objects. + +Get the worker status +^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a GET request to the Replication worker to get the status of the worker service: + +.. code-block: + + GET /worker/service/status + +Get info on requests at various stages of processing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a GET request to the Replication worker to get the information on the requests: + +.. code-block: + + GET /worker/service/requests + +Suspend the worker service +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a PUT request to the Replication worker to suspend the worker service: + +.. code-block: + + PUT /worker/service/suspend + +Resume the worker service +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Drain requests at the worker service +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a PUT request to the Replication worker to drain (stop) all requests in the worker service: + +.. code-block: + + PUT /worker/service/drain + +The operation affects requests that are already in the processing queue or requests that are still +in the input queue waiting to be procesed. The finished requests are not affected by this operation. + + +Reconfigure the worker service +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Controller sends a PUT request to the Replication worker to reconfigure the worker service: + +.. code-block: + + PUT /worker/service/reconfig diff --git a/doc/dev/index.rst b/doc/dev/index.rst index 0fc6b0cc5..6db85a1a0 100644 --- a/doc/dev/index.rst +++ b/doc/dev/index.rst @@ -10,3 +10,4 @@ Developer's Guide quick-start-devel doc scisql + api/index diff --git a/doc/ingest/api/reference/rest/general.rst b/doc/ingest/api/reference/rest/general.rst index 85603dd92..193f869a0 100644 --- a/doc/ingest/api/reference/rest/general.rst +++ b/doc/ingest/api/reference/rest/general.rst @@ -1,3 +1,5 @@ +.. _ingest-general: + General guidelines ==================