Using the REST API =============================== The REST API is used to author new applications that need to interact with Enterprise Gateway. Generally speaking, only the ``/api/kernels`` and ``/api/kernelspecs`` endpoints are used. The ``/api/sessions`` endpoint *can* be used to manage a kernel's lifecycle, but it is not necessary. For example, while the Jupyter Notebook and JupyterLab applications start kernels using ``/api/sessions``, the only interaction they perform with Enterprise Gateway is via the ``/api/kernelspecs`` to retrieve a list of available kernel specifications, and ``/api/kernels`` to start, stop, interrupt and restart a kernel. The "session" remains on the client. General sequence ---------------- Here's the general sequence of events to implement a REST-based application to *discover*, *start*, *execute code*, *interrupt*, and *shutdown* a kernel. To demonstrate each call, we'll use `curl` against a running Enterprise Gateway server at ``http://my-gateway-server.com:8888``. Kernel discovery ~~~~~~~~~~~~~~~~ Issue a `GET` request against the ``/api/kernelspecs`` endpoint to discover available kernel specifications. Each entry corresponds to a ``kernel.json`` file located in a directory that corresponds to the kernel's name. This *name* is what will be used in the subsequent start request. The response is a JSON object where the ``default`` is a string specifying the name of the default kernel. This kernel specification will be used if the start request (e.g., ``POST /api/kernels``) does not specify a kernel name in its JSON body. The other key in the response is `kernelspecs` and consists of a JSON indexed by kernel name with a value corresponding to the corresponding ``kernel.json`` in addition to any *resources* associated with the kernel. These are typically the icon filenames to be used by the front-end application. .. code-block:: console curl http://my-gateway-server.com:8888/api/kernelspecs .. raw:: html
GET /api/kernelspecs response .. code-block:: json { "default": "python3", "kernelspecs": { "python3": { "name": "python3", "spec": { "argv": [ "/usr/bin/env", "/opt/anaconda2/envs/py3/bin/python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python", "interrupt_mode": "signal", "metadata": {} }, "resources": { "logo-32x32": "/kernelspecs/python3/logo-32x32.png", "logo-64x64": "/kernelspecs/python3/logo-64x64.png" } }, "ir": { "name": "ir", "spec": { "argv": [ "R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}" ], "env": {}, "display_name": "R", "language": "R", "interrupt_mode": "signal", "metadata": {} }, "resources": { "kernel.js": "/kernelspecs/ir/kernel.js", "logo-64x64": "/kernelspecs/ir/logo-64x64.png" } }, "spark_r_yarn_client": { "name": "spark_r_yarn_client", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_R_yarn_client/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "SPARK_OPTS": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.sparkr.r.command=/opt/conda/lib/R/bin/Rscript ${KERNEL_EXTRA_SPARK_OPTS}", "LAUNCH_OPTS": "" }, "display_name": "Spark - R (YARN Client Mode)", "language": "R", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy" } } }, "resources": { "kernel.js": "/kernelspecs/spark_r_yarn_client/kernel.js", "logo-64x64": "/kernelspecs/spark_r_yarn_client/logo-64x64.png" } }, "spark_r_yarn_cluster": { "name": "spark_r_yarn_cluster", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_R_yarn_cluster/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "eager" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "SPARK_OPTS": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.am.waitTime=1d --conf spark.yarn.appMasterEnv.PATH=/opt/conda/bin:$PATH --conf spark.sparkr.r.command=/opt/conda/lib/R/bin/Rscript ${KERNEL_EXTRA_SPARK_OPTS}", "LAUNCH_OPTS": "" }, "display_name": "Spark - R (YARN Cluster Mode)", "language": "R", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy" } } }, "resources": { "kernel.js": "/kernelspecs/spark_r_yarn_cluster/kernel.js", "logo-64x64": "/kernelspecs/spark_r_yarn_cluster/logo-64x64.png" } }, "spark_python_yarn_client": { "name": "spark_python_yarn_client", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_python_yarn_client/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "PYSPARK_PYTHON": "/opt/conda/bin/python", "PYTHONPATH": "${HOME}/.local/lib/python3.8/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip", "SPARK_OPTS": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} ${KERNEL_EXTRA_SPARK_OPTS}", "LAUNCH_OPTS": "" }, "display_name": "Spark - Python (YARN Client Mode)", "language": "python", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy" }, "debugger": true } }, "resources": { "logo-64x64": "/kernelspecs/spark_python_yarn_client/logo-64x64.png" } }, "spark_python_yarn_cluster": { "name": "spark_python_yarn_cluster", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_python_yarn_cluster/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "PYSPARK_PYTHON": "/opt/conda/bin/python", "PYTHONPATH": "${HOME}/.local/lib/python3.8/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip", "SPARK_OPTS": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.appMasterEnv.PYTHONUSERBASE=/home/${KERNEL_USERNAME}/.local --conf spark.yarn.appMasterEnv.PYTHONPATH=${HOME}/.local/lib/python3.8/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip --conf spark.yarn.appMasterEnv.PATH=/opt/conda/bin:$PATH ${KERNEL_EXTRA_SPARK_OPTS}", "LAUNCH_OPTS": "" }, "display_name": "Spark - Python (YARN Cluster Mode)", "language": "python", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy" }, "debugger": true } }, "resources": { "logo-64x64": "/kernelspecs/spark_python_yarn_cluster/logo-64x64.png" } }, "spark_scala_yarn_client": { "name": "spark_scala_yarn_client", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_scala_yarn_client/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "__TOREE_SPARK_OPTS__": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} ${KERNEL_EXTRA_SPARK_OPTS}", "__TOREE_OPTS__": "--alternate-sigint USR2", "LAUNCH_OPTS": "", "DEFAULT_INTERPRETER": "Scala" }, "display_name": "Spark - Scala (YARN Client Mode)", "language": "scala", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy" } } }, "resources": { "logo-64x64": "/kernelspecs/spark_scala_yarn_client/logo-64x64.png" } }, "spark_scala_yarn_cluster": { "name": "spark_scala_yarn_cluster", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_scala_yarn_cluster/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "__TOREE_SPARK_OPTS__": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.am.waitTime=1d ${KERNEL_EXTRA_SPARK_OPTS}", "__TOREE_OPTS__": "--alternate-sigint USR2", "LAUNCH_OPTS": "", "DEFAULT_INTERPRETER": "Scala" }, "display_name": "Spark - Scala (YARN Cluster Mode)", "language": "scala", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy" } } }, "resources": { "logo-64x64": "/kernelspecs/spark_scala_yarn_cluster/logo-64x64.png" } } } } .. raw:: html
Kernel start ~~~~~~~~~~~~~~~~ A kernel is started by issuing a ``POST`` request against the ``/api/kernels`` endpoint. The JSON body can take a ``name``, indicating the kernel to start, and an ``env`` JSON, corresponding to environment variables to set in the kernel's environment. In this example, we will start the ``spark_python_yarn_cluster`` kernel with a ``KERNEL_USERNAME`` environment variable of ``jovyan``. .. code-block:: console curl -X POST -i 'http://my-gateway-server.com:8888/api/kernels' --data '{ "name": "spark_python_yarn_cluster", "env": { "KERNEL_USERNAME": "jovyan" }}' .. raw:: html
POST /api/kernels response .. code-block:: json { "id": "f88bdc84-04c6-4021-963d-6811a61eca18", "name": "spark_python_yarn_cluster", "last_activity": "2022-02-12T00:40:45.080107Z", "execution_state": "starting", "connections": 0 } .. raw:: html
Kernel code execution ~~~~~~~~~~~~~~~~~~~~~ Upgrading the connection to a websocket and issuing code against that websocket is currently beyond the knowledge of our maintainers. For this aspect of this discussion we will refer you to our Python `GatewayClient class `_ that we use in our integration tests. .. note:: The name ``GatewayClient`` in our ``enterprise_gateway/client`` subdirectory is not to be confused with the ``GatewayClient`` class defined in the client applications in Jupyter Server and Notebook. In addition, the internal test class ``KernelClient`` is not to be confused with the ``KernelClient`` that lives in the ``jupyter_client`` package. Kernel interrupt ~~~~~~~~~~~~~~~~ A kernel is interrupted by issuing a ``POST`` request against the ``/api/kernels//interrupt`` endpoint. In this example, we will interrupt the ``spark_python_yarn_cluster`` kernel with ID ``f88bdc84-04c6-4021-963d-6811a61eca18`` that was started previously. .. note:: Restarting a kernel is nearly identical to interrupting a kernel; just replace ``interrupt`` in the endpoint with ``restart``. .. code-block:: console curl -X POST -i 'http://ymy-gateway-server.com:8888/api/kernels/f88bdc84-04c6-4021-963d-6811a61eca18/interrupt' An expected response of ``Status Code`` equal ``204`` (No Content) is returned. (The expected response for ``restart`` is ``200`` (OK).) Kernel shutdown ~~~~~~~~~~~~~~~~ A kernel is shutdown by issuing a ``DELETE`` request against the ``/api/kernels/`` endpoint. In this example, we will shutdown the ``spark_python_yarn_cluster`` kernel with ID ``f88bdc84-04c6-4021-963d-6811a61eca18`` that was started previously. .. code-block:: console curl -X DELETE -i 'http://my-gateway-server.com:8888/api/kernels/f88bdc84-04c6-4021-963d-6811a61eca18' An expected response of ``Status Code`` equal ``204`` (No Content) is returned. OpenAPI Specification ~~~~~~~~~~~~~~~~~~~~~ Here's the current `OpenAPI `_ specification available from Enterprise Gateway. An interactive version is available `here `_. .. openapi:: ../../../enterprise_gateway/services/api/swagger.yaml