Building up OpenAPI specs for OpenStack APIs

Date

2024-05-15

Author

Artem Goncharov

Agenda

• Introduction
• OpenStack API "features"
• Designing OpenAPI for OpenStack
• CodeGenerator
• How OpenAPI specs can be used 🦀

What is OpenAPI

The OpenAPI Specification, previously known as the Swagger Specification, is a specification for a machine-readable interface definition language for describing, producing, consuming and visualizing web services.

OpenAPI Example

openapi: 3.1.0
info:
  version: 1.0.0
  title: Sample API
  description: A sample API to illustrate OpenAPI concepts
paths:
  /list:
    get:
      description: Returns a list of stuff
      request:
        headers:
          ..
        content:
          ..
      responses:
        '200':
          headers:
            ..
          content:
            application/json:
              schema:
                description: Stuff items list
                type: array
                items:
                  type: string
                  description: I am the stuff item

What is OpenAPI good for

• Machine readable API contract
• API documentation consistent with the service code
• Generation of API clients
• Enables API security testing
• Ease API integration into 3rd party frameworks (i.e. API Gateway)

OpenStack API

• Every service is exposing their own individual API
• Strongly deviating despite API-SIG effort
• Non-declarative

Dynamic headers

Some services (i.e. Swift) support dynamic (not pre-defined) headers to operate on resource metadata:

$> curl -i $publicURL/marktwain/goodbye -X POST -H "X-Object-Meta-Book: GoodbyeColumbus"

Microversion

“API Microversions” allow changes to the API while preserving backward compatibility. The basic idea is that a user has to explicitly ask for their request to be treated with a particular version of the API.

So breaking changes can be added to the API without breaking users who don’t specifically ask for it.

$> curl -i $publicURL/volumes -X POST -H "OpenStack-API-Version: volume 3.70" ...

RPC style actions

Few OpenStack services rely on the RPC like actions where operation is routed based on the body.

$> curl -i $publicURL/servers/fake_id/action -X POST -d '
    {
        "lock": {"locked_reason": "I don't want to work"}
    }
'

RPC style actions

So now what?

“X-” extensions

OpenAPI allows X-custom extensions

paths:
  /servers:
    put:
      ...
      x-openstack: 
        foo: bar

Dynamic headers

Custom OpenAPI parameter serialization based on regex (similar to parameter serialization)

paths:
  /foo:
    put:
      ...
      parameters:
        - in: header
          name: X-Account-Meta-*
          schema:
            type: string
          x-openstack:
            style: regex

Microversions

JSON schema oneOf with x-openstack extension and custom discriminator (polymorphism)

components:
  schemas:
    VolumesCreateRequest:
      oneOf:
        - $ref: '#/components/schemas/VolumesCreate_30'
        - $ref: '#/components/schemas/VolumesCreate_313'
        - $ref: '#/components/schemas/VolumesCreate_347'
        - $ref: '#/components/schemas/VolumesCreate_353'
      x-openstack:
        discriminator: microversion
    VolumesCreate_313:
      ...
      x-openstack:
        min-ver: 3.13
        max-ver: 3.46
    ...

RPC style Actions

JSON schema oneOf with x-openstack extension and custom discriminator

components:
  schemas:
    server_actions:
      oneOf:
        - $ref: #/components/schemas/action_foo
        - $ref: #/components/schemas/action_bar
      x-openstack:
        discriminator: action
    action_foo:
      ...
      x-openstack:
        action-name: foo
    action_bar:
      ...
      x-openstack:
        action-name: bar

Actions + microversions

Just a combination of above methods.

components:
  schemas:
    server_actions:
      oneOf:
        - $ref: #/components/schemas/action_foo
        - $ref: #/components/schemas/action_bar
      x-openstack:
        discriminator: action
    action_foo:
      oneOf:
        - $ref: #/components/schemas/action_foo_21
        - $ref: #/components/schemas/action_foo_22
      x-openstack:
        action-name: foo
        discriminator: microversion

Constraints

• Strict body definition for certain microversion (no merging or individual parameters)
• Query parameters may be combined for operation marking microversion constraints

...
    parameters:
      new_param:
        in: query
        name: foo_by_bar
        description: Filter foos by bars (new in microversion 2.45)
        schema:
          type: string
        x-openstack:
          min-ver: 2.45

And what’s next?

Let's build OpenAPI specs from service sources

• Inspect source code of services

• Some services have json schema attached to the controllers

• Services use different frameworks (wsgi + routes, pecan, flask, WSME, etc)

• Response descriptions mostly missing

  ==> CodeGenerator

CodeGenerator

• OpenAPI specs by inspecting source code of services
• Rust SDK
• Rust CLI
• Python openstackclient
• Ansible modules
• …

OpenStack supported services

• Nova (compute)
• Cinder (block-storage)
• Glance (image)
• Neutron (network) + plugins
• Keystone (identity)
• Octavia (load-balancer)
• Placement

Nova

• ✅ Routes
• ✅ Request data
• ✅ Microversion info
• ✅ Actions
• ❌ Query parameters
• ❌ Response schemas

Cinder

• ✅ Routes
• ❗ Request data
• ❗ Microversion info
• ✅ Actions
• ❌ Query parameters
• ❌ Response schema

Glance

• ✅ Routes
• ❗ Request data
• ❗ Query parameters
• ❗ Response schema

Keystone

• ✅ Routes
• ❗ Request data
• ❌ Query parameters
• ❌ Response schema

Neutron

• ❗ Routes
• ❗ Request data
• ❗ Query parameters
• ❗ Response schema

Octavia

• ❗ Routes
• ❗ Request data
• ❌ Query parameters
• ✅ Response schema

Challenges

• No naming conventions

• Unified behavior for non-standard service functionality is hard

• Used JSON schema libraries have limited validation of the schema errors

• OpenAPI validation still does not catch all of the JSON schema errors

  => CodeGeneration catches lot of schema errors

"type": "object",
"properties": {
  "schema_version": {"name": {"type": "string"}}
}

Small Survey

Who:

Rust Tooling 🦀💖

https://github.com/gtema/openstack

$> osc --os-cloud devstack compute server list

+------------------+-------+---------+
| id               | name  | status  |
+------------------+-------+---------+
| aaed1db9-***-*** | s**** | ACTIVE  |
| c080769c-***-*** | w**** | ACTIVE  |
| 28aef686-***-*** | y**** | ACTIVE  |
| 3c178e63-***-*** | h**** | ACTIVE  |
| b62f9988-***-*** | 3**** | ACTIVE  |
| 1b129bad-***-*** | s**** | SHUTOFF |
+------------------+-------+---------+

Benchmark

• Scalability
• Sustainability
• Efficiency

Rust tooling

• 10,000loc maintains 500,000+ loc Rust tooling
• Unix philosophy: "do one thing well"
• Awesome performance
• Compiled binary
• Unavoidable consistency (matching to the API functionality)
• ...

Next Steps

• Improve OpenStack pipeline for handling specs
• Improve OpenStack services to add missing schema
• Continue extending generator targets (i.e. TUI, GopherCloud, OpenTofu)
• Central api gateway using OpenAPI specs
  • improved central logging
  • central API metrics capture
  • throttling, Anti-DDoS, etc
• Security scanning of the APIs
• Replace OpenStack API-Ref (documentation)

Thank you

Links:

• https://opendev.org/openstack/codegenerator/
• https://github.com/gtema/openstack
• https://gtema.github.io/openstack-openapi/

Q&A