Optic helps teams write and maintain their first OpenAPI specifications. You don't need to get your team on-board to learn OpenAPI or worry about maintaining 10k line YAML files -- Optic takes care of all of that. Optic manages this process whether your development environment is running locally on your machine or out in the Cloud. No matter where you develop and test, you have confidence that every change to your API is reviewed, approved, and documented before it's released. Your documentation can provide details for your procedures, and assist in your governance goals.
At Optic, we want you to be able to safely and securely store and use your organization's secrets. There are many technologies available, from the cloud provider native solutions such as Azure Key Vault and AWS Key Management Service to open source solutions such as Hashicorp's Vault. Vault is an API-driven tool that secures, stores, and controls authorization for secrets. You can learn more about its claims and how it operates at Hashicorp's Vault project site. The important piece here is that where there's RESTful API driven tooling, there should be Optic. Optic helps me understand not only the APIS I develop, but the APIs I consume. The documentation generated by Optic demonstrates how I do my daily work. It's insight into the practices and procedures my team is currently using, and can help illustrate how we comply with policies such as Information Security.
I set up a quick development environment for Vault to demonstrate how Optic can help document your secret store and make complying with security policies easier. I'll generally follow along with some examples provided in the Vault Getting Started walkthrough. You can follow along as well. The first step is installing Vault. There are many installation options, and I went with:
brew install vault
By default, Vault has a lot of options configured such as enabling TLS. Understanding how Vault is configured out of the box is important when deploying this in any production environment, as Vault is playing a critical role securing secrets. For our demonstration purposes, Vault provides a handy
dev configuration which disables many of the recommended defaults. Secrets are not safe in
dev mode, and I'd recommend against storing valid secrets without a stronger configuration. For our demonstration, though, this will work fine. We'll enable the
dev mode with the
-dev flag, and pass a root token in with
-dev-root-token-id so it's consistent between runs:
vault server -dev-root-token-id=toor -dev
...WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memoryand starts unsealed with a single unseal key. The root token is alreadyauthenticated to the CLI, so you can immediately begin using Vault. You may need to set the following environment variable: $ export VAULT_ADDR='http://127.0.0.1:8200' The unseal key and root token are displayed below in case you want toseal/unseal the Vault or re-authenticate. Unseal Key: 0dNOqFoxFRyx3XuLRTQlraUg6kJvFdlYDUIG+pp6hCI=Root Token: toor Development mode should NOT be used in production installations!
That's was a valid unseal key when I captured it, though it's long been invalidated as it is generated fresh every time we start the
dev mode server. Sealing/Unsealing is not necessary in this configuration, and we can safely ignore it. The only information we need is:
- our root token (
- the Vault API address (
There's several ways to configure Optic, and in this case I'm going to choose to define my inbound and target URLs to match Vault's defaults:
api init --inboundUrl=http://localhost:8000 --targetUrl=http://localhost:8200
Is this your API's root directory? (yes/no): yesWhat is this API named?: Vault[optic] Added Optic configuration to /Users/lou/optic.yml
I'll give my configuration a quick check to make sure it passes:
api check start-proxy
[optic] Testing task start-proxy Given this inboundUrl: http://localhost:8000 ✔ Optic proxy is able to start its proxy here localhost:8000 Given this targetUrl: http://localhost:8200 ✔ Is resolvable from localhost [optic] All Passed! This task is setup properly. Nice work!
Since Vault is already running, the check passes. Hooray! I'll start Optic and we can start experimenting with Vault:
api run start-proxy
[optic] Review the API Diff at http://localhost:34444/apis/1/diffs[optic] Optic is observing requests made to http://localhost:8000
I'm almost ready to send requests through to the Vault service. It's API-driven, and I have many options such as Postman for building request libraries. In a deployed scenario, I'd recommend using a tool like Postman to build a repeatable test collection. For now, I'm going to use the Vault CLI and focus on Optic's documentation process. I'll need to tell Vault to go through Optic so we can observe the traffic with:
Let's say I have an application that needs to be configured with a secret for some reason. Perhaps it needs to connect to another service. I'm currently injecting that secret into my environment somehow, and would rather fetch it from Vault. For a single secret, this may be overkill: you still need to authenticate with Vault in a deployed scenario. I'm forward-thinking, however, and know I'll have a lot of services with many secrets. I also know there are other features in Vault that we'll want to use, and I want to make building and maintaining my documentation part of that process from the start.
The first thing I recommend doing is checking that Vault has started successfully:
Key Value--- -----...Initialized trueSealed false...Version 1.5.0...
I cut out the details to focus on the important parts for my test. The
dev vault is initialized and unsealed (I left version in as well for reference). If you're following along, you want your vault to respond with the same state. Now, let's store our first secret:
vault kv put secret/hello foo=world
Key Value--- -----created_time 2020-09-23T12:59:12.974555Zdeletion_time n/adestroyed falseversion 1
Oops, I forgot: I actually need two keys for this secret:
vault kv put secret/hello foo=world excited=yes
Key Value--- -----created_time 2020-09-23T13:19:04.971314Zdeletion_time n/adestroyed falseversion 2
Did it work? Let's try fetching the secret we stored in
vault kv get secret/hello
====== Metadata ======Key Value--- -----created_time 2020-09-23T13:19:04.971314Zdeletion_time n/adestroyed falseversion 2 ===== Data =====Key Value--- -----excited yesfoo world
Success! I can retrieve just the
excited key as well:
vault kv get -field=excited secret/hello
Now I'll open up the Optic dashboard. My start output earlier reports it's available on
http://localhost:34444/apis/1/diffs. When I open it up, I'll see quite a few items:
There's a few service endpoints fetched often, and some endpoints under
/v1/sys (such as our status check) that we won't care about documenting. We'll ignore those later. For now, let's look into getting our secret data. I'll click through the
GET /v1/secret/data/hello route. After documenting the route in the modals, I see that there's a lot going on under the hood! Optic observed all the traffic, and reduced it to the shape of the data on the right:
An important thing to note here: only the traffic shape is ever stored in the documentation. The traffic itself is never stored in the documentation. Raw traffic is observed and converted to its shape representation locally for documentation and comparison to any existing specification. We don't send your secrets off of your machine.
Going through the requests, Optic has detected our
foo field is always present. It also notices that our
excited field is seen in some requests, and confirms with us:
excitedfield is allowed to be here, and not a mistake.
excitedfield is an optional field, and its omission is not a mistake.
Now I'll commit the changes and accept Optic's default commit message. Optic will let me know that the endpoint is now documented, and the currently observed behavior does not differ from the documented behavior. That makes sense: we just documented all of the observations. I'll document the PUT route as well, and jump ahead to see our generated documentation:
Optic configurations are stored in the
optic.yml file in the root of your API project. One configuration option available is
ignoreRequest. This allows you to define methods and paths to ignore. You can read more about how to ignore requests in our documentation. For now, I'm going to ignore all methods to the noisy routes we've seen. I'll stop my Optic proxy with
CTRL + C to break out of the current session and update my configuration. By default, Optic includes an ignore rule for
OPTIONS requests. When I'm done editing
optic.yml with my editor of choice, my
ignoreRequests section looks like:
ignoreRequests:- OPTIONS (.*)- /socket.io/- /__webpack_hmr- /v1/sys/(.*)
Then, I can restart Optic:
api run start-proxy
Vault should still be running, and a quick invocation of
vault status confirms this. I ran
vault kv get secret/hello, and received my pair of keys as expected. Let's refresh the Optic dashboard at
Success! We can see two things here:
- There are no undocumented URLs visible. Vault is still making calls to service URLs, and I made a call to
/v1/sys/seal-status. The new
ignoreRequestsrules are working as designed. Optic passes through the requests and doesn't show them to us. Nice.
- I requested my secret from
/v1/secret/data/hello, and Optic shows that there are no observed diffs. That means my request and response complies with the currently documented behavior of the API. When my application needs to fetch its secret, I know how to make that request successfully.
Let's say my application is ready to move off of my local machine and into my organization's infrastructure. My organization supports infrastructure under several apps across several environments, and expects a more structured topology as opposed to a flat list of applications. We also require different secrets for different environments. For example, I might need to have a structure that looks more like:
No worries! we can reorganize the layout with a few quick commands. Again, in an actual deployed environment, this process may be very different. I wouldn't recommend having an individual generate secrets via command line for everything, for example. However, we can use the Vault command line for demonstration:
vault kv delete secret/hellovault kv put secret/staging/helloapp foo=region excited=notyetvault kv put secret/prod/helloapp foo=world excited=yesvault kv get secret/staging/helloappvault kv get secret/prod/helloappvault kv get -field=excited secret/staging/helloappvault kv get -field=excited secret/prod/helloapp
To ensure coverage, our list of requests is growing. If we consider environment a parameter, we're also introducing some duplication of requests. This would be a great time to consider something like Postman Environments to parameterize your infrastructure environments, and collections to cover the capabilities you want to deliver with Vault. This will allow you to build repeatable API tests and validate compliance with Optic. For now, let's revisit the Optic dashboard:
Success! We see new undocumented routes for our latest endpoints. Going forward, as our infrastructure evolves and we change both the capabilities and the endpoints to deliver them with Vault, our documentation will stay up to date. When we see changes in behavior, such as new endpoints or new or optional fields, they will be called out by Optic. If there are no undocumented URLs and no endpoint diffs after testing our calls, we can be confident that our secret store is behaving as expected and our applications should be able to fetch their needed configurations in every validated environment.
You can start documenting your API today by setting up Optic with your project. We can help you integrate with your existing documentation tools during our office hours, and would be happy to take feedback on GitHub.