Using the REST API#

The REST API is used to author new applications that need to interact with Enterprise Gateway. Generally speaking, only the /api/kernels and /api/kernelspecs endpoints are used. The /api/sessions endpoint can be used to manage a kernel’s lifecycle, but it is not necessary. For example, while the Jupyter Notebook and JupyterLab applications start kernels using /api/sessions, the only interaction they perform with Enterprise Gateway is via the /api/kernelspecs to retrieve a list of available kernel specifications, and /api/kernels to start, stop, interrupt and restart a kernel. The “session” remains on the client.

General sequence#

Here’s the general sequence of events to implement a REST-based application to discover, start, execute code, interrupt, and shutdown a kernel. To demonstrate each call, we’ll use curl against a running Enterprise Gateway server at http://my-gateway-server.com:8888.

Kernel discovery#

Issue a GET request against the /api/kernelspecs endpoint to discover available kernel specifications. Each entry corresponds to a kernel.json file located in a directory that corresponds to the kernel’s name. This name is what will be used in the subsequent start request.

The response is a JSON object where the default is a string specifying the name of the default kernel. This kernel specification will be used if the start request (e.g., POST /api/kernels) does not specify a kernel name in its JSON body.

The other key in the response is kernelspecs and consists of a JSON indexed by kernel name with a value corresponding to the corresponding kernel.json in addition to any resources associated with the kernel. These are typically the icon filenames to be used by the front-end application.

curl http://my-gateway-server.com:8888/api/kernelspecs
GET /api/kernelspecs response
{
  "default": "python3",
  "kernelspecs": {
    "python3": {
      "name": "python3",
      "spec": {
        "argv": [
          "/usr/bin/env",
          "/opt/anaconda2/envs/py3/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      },
      "resources": {
        "logo-32x32": "/kernelspecs/python3/logo-32x32.png",
        "logo-64x64": "/kernelspecs/python3/logo-64x64.png"
      }
    },
    "ir": {
      "name": "ir",
      "spec": {
        "argv": [
          "R",
          "--slave",
          "-e",
          "IRkernel::main()",
          "--args",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "R",
        "language": "R",
        "interrupt_mode": "signal",
        "metadata": {}
      },
      "resources": {
        "kernel.js": "/kernelspecs/ir/kernel.js",
        "logo-64x64": "/kernelspecs/ir/logo-64x64.png"
      }
    },
    "spark_r_yarn_client": {
      "name": "spark_r_yarn_client",
      "spec": {
        "argv": [
          "/usr/local/share/jupyter/kernels/spark_R_yarn_client/bin/run.sh",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.spark-context-initialization-mode",
          "lazy"
        ],
        "env": {
          "SPARK_HOME": "/usr/hdp/current/spark2-client",
          "SPARK_OPTS": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.sparkr.r.command=/opt/conda/lib/R/bin/Rscript ${KERNEL_EXTRA_SPARK_OPTS}",
          "LAUNCH_OPTS": ""
        },
        "display_name": "Spark - R (YARN Client Mode)",
        "language": "R",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy"
          }
        }
      },
      "resources": {
        "kernel.js": "/kernelspecs/spark_r_yarn_client/kernel.js",
        "logo-64x64": "/kernelspecs/spark_r_yarn_client/logo-64x64.png"
      }
    },
    "spark_r_yarn_cluster": {
      "name": "spark_r_yarn_cluster",
      "spec": {
        "argv": [
          "/usr/local/share/jupyter/kernels/spark_R_yarn_cluster/bin/run.sh",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.spark-context-initialization-mode",
          "eager"
        ],
        "env": {
          "SPARK_HOME": "/usr/hdp/current/spark2-client",
          "SPARK_OPTS": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.am.waitTime=1d --conf spark.yarn.appMasterEnv.PATH=/opt/conda/bin:$PATH --conf spark.sparkr.r.command=/opt/conda/lib/R/bin/Rscript ${KERNEL_EXTRA_SPARK_OPTS}",
          "LAUNCH_OPTS": ""
        },
        "display_name": "Spark - R (YARN Cluster Mode)",
        "language": "R",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy"
          }
        }
      },
      "resources": {
        "kernel.js": "/kernelspecs/spark_r_yarn_cluster/kernel.js",
        "logo-64x64": "/kernelspecs/spark_r_yarn_cluster/logo-64x64.png"
      }
    },
    "spark_python_yarn_client": {
      "name": "spark_python_yarn_client",
      "spec": {
        "argv": [
          "/usr/local/share/jupyter/kernels/spark_python_yarn_client/bin/run.sh",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.spark-context-initialization-mode",
          "lazy"
        ],
        "env": {
          "SPARK_HOME": "/usr/hdp/current/spark2-client",
          "PYSPARK_PYTHON": "/opt/conda/bin/python",
          "PYTHONPATH": "${HOME}/.local/lib/python3.8/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip",
          "SPARK_OPTS": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} ${KERNEL_EXTRA_SPARK_OPTS}",
          "LAUNCH_OPTS": ""
        },
        "display_name": "Spark - Python (YARN Client Mode)",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy"
          },
          "debugger": true
        }
      },
      "resources": {
        "logo-64x64": "/kernelspecs/spark_python_yarn_client/logo-64x64.png"
      }
    },
    "spark_python_yarn_cluster": {
      "name": "spark_python_yarn_cluster",
      "spec": {
        "argv": [
          "/usr/local/share/jupyter/kernels/spark_python_yarn_cluster/bin/run.sh",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.spark-context-initialization-mode",
          "lazy"
        ],
        "env": {
          "SPARK_HOME": "/usr/hdp/current/spark2-client",
          "PYSPARK_PYTHON": "/opt/conda/bin/python",
          "PYTHONPATH": "${HOME}/.local/lib/python3.8/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip",
          "SPARK_OPTS": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.appMasterEnv.PYTHONUSERBASE=/home/${KERNEL_USERNAME}/.local --conf spark.yarn.appMasterEnv.PYTHONPATH=${HOME}/.local/lib/python3.8/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip --conf spark.yarn.appMasterEnv.PATH=/opt/conda/bin:$PATH ${KERNEL_EXTRA_SPARK_OPTS}",
          "LAUNCH_OPTS": ""
        },
        "display_name": "Spark - Python (YARN Cluster Mode)",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy"
          },
          "debugger": true
        }
      },
      "resources": {
        "logo-64x64": "/kernelspecs/spark_python_yarn_cluster/logo-64x64.png"
      }
    },
    "spark_scala_yarn_client": {
      "name": "spark_scala_yarn_client",
      "spec": {
        "argv": [
          "/usr/local/share/jupyter/kernels/spark_scala_yarn_client/bin/run.sh",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.spark-context-initialization-mode",
          "lazy"
        ],
        "env": {
          "SPARK_HOME": "/usr/hdp/current/spark2-client",
          "__TOREE_SPARK_OPTS__": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} ${KERNEL_EXTRA_SPARK_OPTS}",
          "__TOREE_OPTS__": "--alternate-sigint USR2",
          "LAUNCH_OPTS": "",
          "DEFAULT_INTERPRETER": "Scala"
        },
        "display_name": "Spark - Scala (YARN Client Mode)",
        "language": "scala",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy"
          }
        }
      },
      "resources": {
        "logo-64x64": "/kernelspecs/spark_scala_yarn_client/logo-64x64.png"
      }
    },
    "spark_scala_yarn_cluster": {
      "name": "spark_scala_yarn_cluster",
      "spec": {
        "argv": [
          "/usr/local/share/jupyter/kernels/spark_scala_yarn_cluster/bin/run.sh",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.spark-context-initialization-mode",
          "lazy"
        ],
        "env": {
          "SPARK_HOME": "/usr/hdp/current/spark2-client",
          "__TOREE_SPARK_OPTS__": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.am.waitTime=1d ${KERNEL_EXTRA_SPARK_OPTS}",
          "__TOREE_OPTS__": "--alternate-sigint USR2",
          "LAUNCH_OPTS": "",
          "DEFAULT_INTERPRETER": "Scala"
        },
        "display_name": "Spark - Scala (YARN Cluster Mode)",
        "language": "scala",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy"
          }
        }
      },
      "resources": {
        "logo-64x64": "/kernelspecs/spark_scala_yarn_cluster/logo-64x64.png"
      }
    }
  }
}

Kernel start#

A kernel is started by issuing a POST request against the /api/kernels endpoint. The JSON body can take a name, indicating the kernel to start, and an env JSON, corresponding to environment variables to set in the kernel’s environment.

In this example, we will start the spark_python_yarn_cluster kernel with a KERNEL_USERNAME environment variable of jovyan.

curl -X POST -i 'http://my-gateway-server.com:8888/api/kernels' --data '{ "name": "spark_python_yarn_cluster", "env": { "KERNEL_USERNAME": "jovyan" }}'
POST /api/kernels response
{
  "id": "f88bdc84-04c6-4021-963d-6811a61eca18",
  "name": "spark_python_yarn_cluster",
  "last_activity": "2022-02-12T00:40:45.080107Z",
  "execution_state": "starting",
  "connections": 0
}

Kernel code execution#

Upgrading the connection to a websocket and issuing code against that websocket is currently beyond the knowledge of our maintainers. For this aspect of this discussion we will refer you to our Python GatewayClient class that we use in our integration tests.

Note

The name GatewayClient in our enterprise_gateway/client subdirectory is not to be confused with the GatewayClient class defined in the client applications in Jupyter Server and Notebook. In addition, the internal test class KernelClient is not to be confused with the KernelClient that lives in the jupyter_client package.

Kernel interrupt#

A kernel is interrupted by issuing a POST request against the /api/kernels/<kernel_id>/interrupt endpoint.

In this example, we will interrupt the spark_python_yarn_cluster kernel with ID f88bdc84-04c6-4021-963d-6811a61eca18 that was started previously.

Note

Restarting a kernel is nearly identical to interrupting a kernel; just replace interrupt in the endpoint with restart.

curl -X POST -i 'http://ymy-gateway-server.com:8888/api/kernels/f88bdc84-04c6-4021-963d-6811a61eca18/interrupt'

An expected response of Status Code equal 204 (No Content) is returned. (The expected response for restart is 200 (OK).)

Kernel shutdown#

A kernel is shutdown by issuing a DELETE request against the /api/kernels/<kernel_id> endpoint.

In this example, we will shutdown the spark_python_yarn_cluster kernel with ID f88bdc84-04c6-4021-963d-6811a61eca18 that was started previously.

curl -X DELETE -i 'http://my-gateway-server.com:8888/api/kernels/f88bdc84-04c6-4021-963d-6811a61eca18'

An expected response of Status Code equal 204 (No Content) is returned.

OpenAPI Specification#

Here’s the current OpenAPI specification available from Enterprise Gateway. An interactive version is available here.

GET /api#

Get API info

Status Codes:

  • 200 OK – Returns information about the API

Response JSON Object:

  • gateway_version (string) –

  • version (string) –

GET /api/swagger.yaml#

Get API info

Status Codes:

  • 200 OK – Returns a swagger specification in yaml

GET /api/swagger.json#

Get API info

Status Codes:

  • 200 OK – Returns a swagger specification in json

GET /api/kernelspecs#

Get kernel specs

Query Parameters:

  • user (string) – When present, kernelspec results will be filtered based on the configured authorization of specified value.

Status Codes:

  • 200 OK – If no query parameter is specified, all kernel specs will be returned; otherwise the result set is filtered based on the query parameter.

Response JSON Object:

  • default (string) – The name of the default kernel.

  • kernelspecs (object) –

GET /api/kernels#

List the JSON data for all currently running kernels

Status Codes:

  • 200 OK – List of running kernels

  • 403 Forbidden – This method is not accessible when EnterpriseGatewayApp.list_kernels is False.

Response JSON Object:

  • [] (any) – Kernel information

POST /api/kernels#

Start a kernel and return the uuid

Request JSON Object:

  • env (object) – A dictionary of environment variables and values to include in the kernel process - subject to filtering.

  • name (string) – Kernel spec name (defaults to default kernel spec for server)

Status Codes:

  • 201 Created – The metadata about the newly created kernel.

  • 403 Forbidden – The maximum number of kernels have been created.

Response Headers:

  • Location – Model for started kernel

GET /api/kernels/{kernel_id}#

Get kernel information

Parameters:

  • kernel_id (string) – kernel uuid

Status Codes:

  • 200 OK – Information about the kernel

DELETE /api/kernels/{kernel_id}#

Kill a kernel and delete the kernel id

Parameters:

  • kernel_id (string) – kernel uuid

Status Codes:
GET /api/kernels/{kernel_id}/channels#

Upgrades the connection to a websocket connection.

Parameters:

  • kernel_id (string) – kernel uuid

Status Codes:

  • 200 OK – The connection will be upgraded to a websocket.

POST /kernels/{kernel_id}/interrupt#

Interrupt a kernel

Parameters:

  • kernel_id (string) – kernel uuid

Status Codes:
POST /kernels/{kernel_id}/restart#

Restart a kernel

Parameters:

  • kernel_id (string) – kernel uuid

Status Codes:

  • 200 OK – Kernel interrupted

Response Headers:

  • Location – URL for kernel commands

GET /api/sessions#

List available sessions

Status Codes:

  • 200 OK – List of current sessions

  • 403 Forbidden – This method is not accessible when the kernel gateway when the list_kernels option is False.

Response JSON Object:

  • [].id (string) –

  • [].kernel (any) – Kernel information

  • [].name (string) – name of the session

  • [].path (string) – path to the session

  • [].type (string) – session type

POST /api/sessions#

Create a new session, or return an existing session if a session of the same name already exists.

Request JSON Object:

  • id (string) –

  • kernel (any) – Kernel information

  • name (string) – name of the session

  • path (string) – path to the session

  • type (string) – session type

Status Codes:
Response Headers:

  • Location – URL for session commands

Response JSON Object:

  • id (string) –

  • kernel (any) – Kernel information

  • name (string) – name of the session

  • path (string) – path to the session

  • type (string) – session type

GET /api/sessions/{session}#

Get session

Parameters:

  • session (string) – session uuid

Status Codes:
Response JSON Object:

  • id (string) –

  • kernel (any) – Kernel information

  • name (string) – name of the session

  • path (string) – path to the session

  • type (string) – session type

PATCH /api/sessions/{session}#

This can be used to rename the session.

Parameters:

  • session (string) – session uuid

Request JSON Object:

  • id (string) –

  • kernel (any) – Kernel information

  • name (string) – name of the session

  • path (string) – path to the session

  • type (string) – session type

Status Codes:
Response JSON Object:

  • id (string) –

  • kernel (any) – Kernel information

  • name (string) – name of the session

  • path (string) – path to the session

  • type (string) – session type

DELETE /api/sessions/{session}#

Delete a session

Parameters:

  • session (string) – session uuid

Status Codes:

  • 204 No Content – Session (and kernel) were deleted

  • 410 Gone – Kernel was deleted before the session, and the session was not deleted