API Evangelist

Gathering My Thoughts On Api Discovery

Thu, 11 Feb 2021 00:00:00 +0000

published: true

title: ‘Gathering My Thoughts on API Discovery’

image: https://kinlane-productions2.s3.amazonaws.com/postman-network.png

I am working to load up all my API discovery experiences into my head for some upcoming conversations. So I sat down and pulled together a summary of my API discovery research to date to help refresh my memory of what has happened and how we got here. API discovery is the one of the few layers of the API space that I am personally committed to helping move things forward and being able to see all the moving parts together helps me keep doing this. Let’s take a stroll through my memory of the evolution of API discover over the last fifteen years so that I can speak more coherently about all of this with a variety of folks.

ProgrammableWeb

ProgrammableWeb was the first source of being able to discover API, and in 2020 it is still the place you go to find APIs. Really not much has changed in the last fifteen years with ProgrammableWeb except for the owners and operators, and the look and feel of the site. It is still where you go to look for new and existing APIs, and where you find APIs when Googling.

I have fond memories of writing for ProgrammableWeb, and the site is still a great source of information for me, but I am left frustrated that PW hasn’t moved forward the API discovery conversation in any interesting ways over the years. I just think it is a missed opportunity, and reflects much about the API space that I think holds us all back.

Mashape -> Rapid API

After ProgrammableWeb, the next evolution in the API space when it came to API discovery was Mashape, which is now known as Rapid API. The API marketplace was born out of the age of API management and provides basic management capabilities alongside with API discovery services. Providing a pretty rich set of APIs you can search and onboard with using the Rapid API marketplace.</p>

Mashape and Rapid API definitely moved forward the API discovery conversation a bit, but much like ProgrammableWeb, it stopped there. RapidAPI is definitely giving ProgrammableWeb a run for their money when it comes to the SEO game, but there really hasn’t been much out of RapidAPI beyond the fundamentals of a simple API marketplace—-that is fine, but I’m always looking for forward motion.

US Federal Government Data.json Index

Next I am going to take a little detour, but it relates to API discovery, I promise. In 2013 I went to work for the Obama administration to work on helping federal agencies publish their public data assets using APIs. I was dedicated to the Department of Veterans Affairs, but also spent time working with other agencies to follow the presidential mandate to publish what is called a data.json file in the root of the web site domains for the 15 executive agencies—-you can still find the fifteen files available today.

I have setup a workspace to pull all fifteen of the data.json files and I am working to process them to see what has been published, and understand the state of the work that I began back in 2013. I learned a lot about data discovery and the realities of doing it within large bureaucracies as part of this work, something that I brought home with me after my tour was done and began applying to what I wanted to see when it comes to mainstream API discovery.

APIs.json / APIs.yaml

Shortly after leaving Washington DC I created a new API discovery format with Steve Willmott from 3Scale called APIs.json. Like the data.json file, APIs.json would provide an discovery format for APIs, but I would evolve to better fit what I felt was needed to help move the API discovery conversation forward inside and outside of government. Providing a way to index individual APIs, collections of APIs, workflows, and other ways we are needing to discover and put APIs to work in automated and manual ways.

I am about to move the specification forward to the next version, and will keep working to evolve the specification. I have over 40K APIs.json files for companies, organizations, institutions, and government agencies I have identified of having an API, working with APIs, or being suspect of having an API. I will keep working to iterate upon each of the entries I have in the list as I continue to expand the ways in which I am crawling and discovering APIs.

APIs.io

When we launched APIs.json we also launched APIs.io, the first API search engine. We wanted an implementation of the API discovery specification to show what was possible pushing API providers to learn more about, and begin publish the specification as part of their operations. APIs.io was also beginning to poll for updates to APIs.json files it indexed, as well as beginning to consider how to search for new ones. Making it the first API search engine that allows you to search for APIs, while also looking for updates, and expanding the catalog in an automated way.

Sadly I know that APIs.io isn’t maintained. People are still adding APIs to it, but if the site goes down I am guessing there is nobody home to get back up and running. The site got lost in the shuffle of Red Hat acquiring 3Scale and then IBM acquiring Red Hat. While the site isn’t a living part of the API discovery conversation it still plays a role in shaping the discussion.

API Specifications

Alongside the evolution of ProgrammableWeb, RapidAPI, and the introduction of APIs.json and APIs.io, a variety of specifications emerged to help improve the API lifecycle, which all contributed to making APIs a little more tangible and ultimately discoverable. Each of these API specifications have made it easier to find APIs, whether you are looking on the web, via Github, or somewhere on your local network.

OpenAPI (Swagger) - OpenAPI provides an artifact that is discoverable and is organized by a variety of marketplaces, directories, and hubs to make them more discoverable.
API Blueprint - While API Blueprint hasn’t one the API specification war, it did significantly contribute to the API discovery conversation because .apib extensions are easy to find.
RAML - Like API Blueprint, RAML files are pretty easy to discover and find on Github and powering documentation, helping make APIs much more discoverable.
Postman Collections - The Postman platform uses collections to define APIs, and with over 13M developers making collections, they’ve become a great way to find and share APIs.

API specifications provide a pretty critical building block when it comes to API discovery. These machine readable specifications describe the surface area of an API, while providing a title, description, and other metadata that help make an API more discoverable. Organizing API specifications into a marketplace, directory, or hub would turn out to be the next definitive step in the API discovery conversation.

SwaggerHub

After Smartbear put Swagger into the Linux Foundation as OpenAPI, they launched SwaggerHub to help API providers publish their OpenAPIs, making them available for discovery. The platform does as it says, provides a hub for Swagger, and now OpenAPI. Allowing developers to publish and search for different types of APIs, while also being able to edit and orchestrate using them across the API lifecycle.

SwaggerHub is definitely an API discovery solution, but it focuses on it from a very different postition. It is more about editing and designing APIs, then making them available across multiple stops along the API lifecycle, including discovery. Building on top of the leading API specifications, but focusing on how it will be put to work across the API lifecycle.

APIs.guru

Similar to SwaggerHub, another more open source approach emerged with APIs.guru, which also aggregates Swagger and OpenAPI definitions, but does it as a GitHub repository, with a simple search website built on top. Providing a rich catalog of OpenAPI definitions for some of the most known APIs. Providing a single place you can go to download or fork the OpenAPI for a variety of APIs, making search dead simple.

I like the APIs.guru approach. API discovery on Github makes sense to me. GitHub isn’t just for managing code and provides a wealth of features you can use to move forward different APIs using OpenAPI and other specifications. You can fork the entire APIs.guru OpenAPI catalog, making it pretty portable and forkable API discovery solution that you can run anywhere.

Postman Network

As all of these other things were happening, Postman published their own API directory that is called the Postman API Network. Providing a listing of different APIs which are defined by Postman’s own API specification called collections. Postman users can publish their collections to the API network, then consumers can browse APIs by category, or search for APIs by keyword.

With the latest release of the Postman platform you can now find APIs defined by OpenAPI, RAML, or GraphQL, as well as collections, monitors, mock servers, and other components in publish workspaces. Think Github repositories but designed for APIs. Additionally the search for the Postman network has evolved to provide a more detailed look at public APIs, but also APIs you have available privately within personal or team workspaces.

More API Catalogs

Then we began to see more API directories and catalogs emerge. There has been a lot more than these which ultimately have gone away, but this shows a nice cross section of the types of API discovery solutions that have emerged--helping all of us find the APIs we are needing in our applications.

Most API directories launch, then receive waves of updates, then go dormant or disappear entirely. I’ve launched several directories myself, only to take them down once they go stale. Keeping API catalogs fresh and up to date takes a lot of work and there really isn’t much money to be made in doing--at least not yet.

API Discovery with a Purpose

More recently we’ve begun to see a fresh wave of API discovery solutions emerge, but this crop of solutions isn’t just about simply finding APIs. It is about finding APIs so that you can meet some other more pressing needs like security or compliance. Here are handful of the new types of approaches to finding APIs, helping us make sense of our very abstract and invisible infrastructure we are increasingly dependent on that is growing exponentially around us.

You see a whole mix of why we need to discover APIs represented here. I feel like search to just discover an API isn’t enough to take API discovery to the next level. We are needing other reasons to discover the API infrastructure that is emerging all around us. APIs are often growing so fast that we can’t keep up with documentation and other more traditional ways in which we see and discover APIs.

Just Google It

In reality, most APIs are discovered by a simple Google search. It is something that I think will always remain a reality, and something I think Google will continue to invest in to help us find the APIs we need. However, this will only help us surface the public APIs we are looking for. We will need solutions that help us find public APIs, but also the private and partner ones we depend on which aren't discoverable using a Google search.

The Joy of Finding New APIs

There really aren’t many ways to be introduced to new APIs on a regular basis. You can learn new ones by reading Techcrunch, tuning into the right slices of the Twitterverse or Reddit, but I’d say that Product Hunt. You can also tune into ProgramambleWeb and get introduced to new APIs using their steady drip of new APIs, but I ultimately wish that there were more ways of being introduced APIs, but is probably the way that is the least likely to be monetizable.

The Further Semantics of Discovery

Adding another layer to this discussion, driving how APIs are discovered via Google and other search engines, are some machine readable building blocks that help us better describe the resources we are making available via APIs. There are a couple of ways we can markup, add metadata, and provide more detail that help make our APIs more discoverable by default.

The challenge with all of these building blocks is that they take a lot of work to add to each of our APIs, and it is highly unlikely API providers are going to find the time to do it. So it is up to service and tooling providers to mark things up, and augment API definitions with common elements like Schema.org, JSON-LD, and other rich formats that help us make sense of the digital resources and capabilities we are making available.

A Peak At Laneworks API Discovery

I have made several attempts to provide an API discovery solution over the years, and APIs.json only recently started to move forward again. I have a few other solutions that have come and gone, with the most recent evolution being part of my internal platform I have built which I simply call Laneworks. This latest investment is all about API discovery and finding new and interesting APIs on the web and investing them for inclusion into a directory.

OpenAPI Search - I crawl for Swagger and OpenAPI definitions to discover new APIs.
Postman Collection Search - I crawl for Postman collections to discover new APIs.
Vocabulary / Dictionary - I have a robust dictionary for seeding my searches for APIs.
Domains - I try to organize key words and phrases into different domains I can work within.
GitHub Search - I apply my dictionary to searching across Github for API artifacts.
Bing Search - I apply my dictionary to searching for APIs using the Bing search API.

I am retiring the v1 of my Laneworks approach to automating API discovery and I am reworking it to operate within a public workspace. I moved my couple thousand of OpenAPIs and resulting collection to a couple thousand public workspaces. Then I’ll be reworking this automation as Postman collections that run as monitors on a schedule, allowing me to discover, harvest, parse, rank, and organize via workspaces for discovery. When I have this on it tends to uncover a lot of interesting APIs, which can be easily catalogued, but the final touches on polishing and making available has always taken time, something I haven’t historically had--who knows what the future will hold.

New API Discovery Tooling

I put this at the end because it just happened a couple weeks ago, but it is a very promising sign of progress in the API discovery conversation. It is a new open source discovery solution that is the most sophisticated open source approach I’ve seen to helping solve the discovery of both public and private APIs.

Google API Registry

I always wondered why Google hand’t gotten into the API discovery game. They own the search market, and they have had a discovery solution for their own stack of APIs, but this open source approach shows the potential of something more. I still have to play with the Google API Registry to understand what it does, but I think it is significant enough to include in this API discovery narrative.

Important Questions to Ask

The biggest challenge with API discovery to date is that there are many dimensions to grapple with. It isn’t just about search or discovery of APIs. It is about being found. It is about how well you describe what you do. It depends on who you are and what your motivations are. There are a handful of questions that I ask to help me understand the multiple dimensions of the API discovery conversation so that I can see it all.

Who is searching? Provider or Consumer?
Where are they searching? Publicly or Privately?
Why are they searching for APIs?
Is human searching, or is it a computer searching?
What is the popularity, quality, reliability of the API?

The answers to these questions will shift what API discovery is in any given moment. There are many other things to consider, but these areas reflect the common ways I have seen muddy the API discovery waters over the years. Making it something that is difficult to actually provide an easily solution for or explain to folks what it is all about. Honestly, I need to regular checkins on what is happening, and do regular refreshes so that I can even make some sense of things. Sometimes I feel like nothing has happened when it comes to API discovery, but once I do refreshes like this I realize just how much has been happening—-it is just more incremental.

API Discovery Drivers

Now that I have all of this loaded up in my head I wanted to think a little bit about what drives API discovery. I want to understand the motivations behind why people are looking to discover APIs, and why those who own APIs are looking to have them discovered, and what is empowering these searches. I am looking to understand where in the lifecycle API discovery is needed most. Scanning down this post, here are the different elements I am thinking of as I try to make sense of where we stand with API discovery in 2021.

Search - Having a search box handy is central to API discovery, whether that is in application, via local network, or on the open web using a search engine.
Specifications - OpenAPI, AsyncAPI, Postman Collections, and JSON Schema will continue to play central role in the API discovery combination.
Vocabulary - Having a formal vocabulary for discovering new APIs in an automated way, while also using it to markup meta data for APIs is important.
Domains - Bounded contexts are critical for not just designing APIs, but also organizing and making them available as part of API discovery.
Semantics - APIs need to be augmented with rich semantics that help us understand the resources and capabilities being made available to users.
Ease - API discovery has to be easy no matter who you are, and something that has to consider personas, context, and much more under the hood.
Security - You can’t secure what you don’t know about, so security will continue to be one of the most important reasons why we do API discovery.
Change - Managing change across the enterprise is the biggest challenge teams face, and API discovery needs to consider how things change or not.
Traceability - We are in desperate need of the ability to trace out what is going on with the big ball of API yarn we’ve spread across our organizations.
Vulnerabilities - Finding the vulnerabilities across our infrastructure before bad people do is a top priority for the API discovery conversation.
Compliance - Being compliant when it comes to our API operations is a central concern for those in charge, especially as things keep expanding.
Regulation - When it comes to GDPR and other regulations emerging, we are increasingly going to be able to discovery exactly the resources we need.
Monitoring - Including monitoring data for all APIs as part of API discovery plays a big role in understanding the uptime and availability of resources.
Testing - Augmenting monitoring data, a range of testing data needs to be included as part of the search and discovery of each and every API.
Reliability - You want to find the APIs that are reliable when searching for 3rd party or public APIs, and find those that aren’t reliable internally.
Quality - The overall quality of each API across many dimensions has to be included in the overall API discovery index that powers what we discover.
Automatic - API discovery in all forms has to be automated. It is clear that we can’t count on humans to catalog and make APIs available for discovery.
Public - Despite concerns public APIs will continue to dominate and driver the API discovery conversation because it is the easiest to reference.
Private - Private API discovery is where we are seeing the most growth and demand for API discovery solutions in the last five years.
Identity - Who you are and what credentials possess, and the policies applied to those credentials will keep driving the API discovery conversation.
Authentication - Being able to apply or at least consider authentication at the API discovery layer will help reduce friction at the discovery layer.
Editor - Being able to edit API definitions, metadata, and other elements used in discovery seems to be a significant part of how we discover and work with APIs.
Rating - We are going to need a multi-part sophisticated approach to rating APIs so that we can focus on just what matters to use when it comes to discovery.
Forkability - The ability to fork the APIs we are discovering, and sort and rank based upon what we’ve forked and merged will continue to drive search.
SEO - The search engine optimization game will continue to drive the API discovery conversation, and keep shaping how we describe the APIs we have.
Content - The narrative for each API and it’s consumes and applications play a big role in helping us discover what is going on around each API.

I will have to simmer on these elements before writing much more. There is definitely a lot more to API discovery than organizing APIs into a catalog and providing a search mechanism, but I’m still not sure what the next evolution of API discovery will be. I definitely feel a lot better when it come to thinking nothing has happened with API discovery in the last decade-—plenty has happened. As always in the API space the trick is always about aggregating everything together and reading the tea leaves to figure out what is happening, and considering what the needs of API publishers and consumers are--which is what this blog post is all about.

I am always skeptical of my views of API discovery. I am an analyst, as well as a publisher and consumer of APIs. I am also API obsessed. Which puts me into a whole different category when it comes to wanting to find APIs. I don’t think the average business or technical person cares about APIs—-they are just trying to get work done in their worlds. I think search is central to API discovery, but there are a number of other mechanisms at work that will make that search relevant or not relevant to whole mix of different personas who have often competing motivations for why they want to find APIs, or have their APIs found. In the end, I think the default mode of API discovery is still just “Googling it”. I am not sure if this is more about web search and discovery than it is about API discovery, or there is something deeper to consider. Regardless, I think API discovery still needs to heavily consider this when it comes to API discovery for public APIs, but when it comes to private APIs I have other ideas-—more to come..

Api Storytelling With Mike And Aidan

Thu, 11 Feb 2021 00:00:00 +0000

published: true

title: ‘API Storytelling with Mike and Aidan’

image: https://kinlane-productions2.s3.amazonaws.com/api-storytelling-with-mike-and-aidan.png

If you have followed my work you know that I like telling stories. Stories are the single most important tool in my Chief Evangelist and API Evangelist toolbox. None of the code matters in my opinion, and the stories surrounding the code is what makes the actual impact on our reality. The stories we tell each other, as well as the stories we tell ourselves. Everything in the world of technology and APIs is made up of stories. I am hovering around 5000 stories here on API Evangelist, which reflects everything I am when it comes to API Evangelist. Stories are how I make sense of the API world that has unfolded in the last twenty years. Storytelling is how I work through all of the things I do not understand, moving everything API from a very abstract realm into something that is more tangible, visible, and sometimes more meaningful to me in the real world. I couldn’t have done any of this without stories.

Throughout my storytelling as the API Evangelist I have also taught myself to work through my own personal baggage over at kinlane.com. Applying the same methodology to my past and figuring out who I am today. API Evangelist has always been very much a performance, but the storytelling I engaged in as part of the performance went much deeper than being just a business production. Over the years, both of these streams of storytelling (apievangelist.com & kinlane.com) have influenced others in the world, having an impact on their storytelling, or help them embark on their own storytelling journey. I’ve learned a lot from these engagements, and made a lot of friends around the world because of a shared love for storytelling. This shared storytelling journey has recently manifested itself as a video conversation between Mike Amundsen (@mamund) and Aidan Cunniffe (@aidandcunniffe), which we are simply calling API Storytelling. We only have one discussion under our belt and we aren’t quite sure where we are going with this, but the result for me is a nourishing 60 minutes of storytelling goodness.

Why did we choose APIs? We talk about it a little bit in the video, but APIs are something that can be associated with almost every aspect of our world, making for a pretty good switchboard for endless storytelling. The three of us are all API storytellers, but we also have witnessed first hand how storytelling helps you personally grow and make sense of the world as well as ourselves. I love hearing how Mike and Aidan wrestle with storytelling and how it works in their world. Listening to how they use storytelling to break down their business worlds, but also their personal life. We don’t have a plan for our API storytelling conversations. We are just showing up with our storytelling bag and sharing little bits this and that and seeing where it all goes. If you watch the first edition of this API storytelling conversation you’ll see that we talk about what we’d like to see this turn into, as well as just the wandering through each of our views on storytelling.

We are going to do another API Storytelling session tomorrow. I am curious to see where we go with things. I can’t articulate just how important storytelling is in my life. I know it sounds corny, but it is true. It is what keeps me going in this crazy world. Stories are really the keys to everything for me. I am also very happy to have others to share this with. I am fascinated to learn how Mike and Aidan approach their version of telling stories to keep moving forward. I am not sure if this is anything that others will want to tune into but it is something that I am glad to have happening right now. I feel like there needs to be a while lot of storytelling going on in this moment, and somehow this is what will elevate us out of this very strange set of times we find ourselves in.

Making Sense Of The Different Types Of Api Testing

Sat, 30 Jan 2021 00:00:00 +0000

published: true

title: ‘Making Sense of the Different Types of API Testing’

image: https://s3.amazonaws.com/kinlane-productions2/algorotoscope-master/america-immigration_dumping-ground-gauge-on-aircraft-carrier.jpg

I have to admit something. I don’t fully grasp the entire landscape of API testing. I mean, I have a pretty decent awareness and experience in testing APIs, but I can’t close my eyes and coherently break down each layer or type of testing, let alone competently navigate the many different types of services and tooling available on the market for API testing. Like most other dimensions of the API universe the content and guidance on the subject of API testing is all over the place. After a couple of hours Googling and making my way through my bookmarks, I uncovered a pretty expansive dictionary for what I’d consider to be under the umbrella of API testing.

First off, I have to talk about API testing in the most general sense before I get into the more and less formal API testing approaches. It was a dimension of API testing I hadn’t considered before I began working at Postman, but API testing meaning that I am just kicking the tires on a API, learning what it is all about, and how you use it. This is what the majority of Postman’s 13 million users are doing when they say they are testing an API with the Postman platform. THis experimental and curious approach to API testing provides a great onramp to the world of API publishing and consumption, but does little to help us articulate what API testing is or isn’t.

Beyond just “testing out” an API, from what I can gather across about 25 separate API testing articles, there are 25 different layers to the API testing onion:

Validation Testing - I am guessing a general type of validation — seems pretty broad.
Functional Testing - Adopted as part of code testing practices, applied to unit of code.
Component Testing - Similar to functional, but testing based upon small unit of compute.
User Interface Testing - Testing of the API in the context of the end user interface.
Load Testing - Seeing how many requests each API is able to handle in a timeframe.
Performance Testing - Overlapping with load testing, but benchmarking the API performance.
Reliability Testing - Understanding the overall reliability of each individual API.
Runtime Error Detection - Focusing specifically on errors encountered at runtime.
Security Testing - A broad umbrella for testing the layers of API security that is present.
Authorization Tests - Focused on testing the authentication for each individual API.
Penetration Testing - Seeing you can actually get into an API through alternate ways.
Fuzz Testing - Blasting an API with garbage, noise, and other junk to see what it does.
Interoperability Testing - Testing how well an API plays with other external APIs.
Integration Testing - Testing an API based upon its dependency on external APIs.
Discovery Testing - Understanding the discoverability of an API as part of operations.
Usability Testing - Testing an API from the perspective of how usable it is to consumers.
Behavioral Testing - Driving testing based upon external views of what is expected of the API.
Acceptance Tests - Tested defined by the consumer based upon what is meaningful to them.
End-to-End Testing - Testing the entire API stack from backend to front, and back again.
Contract Testing - Deriving specific business contracts for an API and testing them out.
Scenario Testing - Building tests that reflect specific business scenarios that are relied upon.
Workflow Testing - Another way to define a test for a series of API calls that reflect a business need.
Omni-Channel Tests - Testing out multiple channels of consumption like web, mobile, and devices.
Documentation Testing - Verifying that an API possess valid and up to date documentation.
Data-Driven Testing - Relying on data to define and drive the testing of each API.

Don’t count on any of these descriptions being accurate. They are more about be trying to make sense of each one. I will be doing more research into what each of these mean to services and tooling providers, but I wanted to try and at least get all of them on the table, and try to squint and see if I could find any interesting patterns across them. There are a mix of personas, intentions, and perceptions about where the value and pain lie in the API stack present in this API testing vocabulary. Seeing all of these types of API testing really doesn’t help me understand much, or be able to better articulate it to my readers and Postman customers. It really is just a dizzying amount of activity going on across this slice of the API universe.

Considering API Governance as Part of API Testing

Everything I have covered so far is about testing API instances. Making sure each individual is doing what it should be.However, over the last decade a new dimension of testing has emerged that isn’t about testing each individual API and it’s requests and responses, but testing the surface area of the API, making sure it is consistent with the overall organizational strategy or industry standards. API governance tends to focus on the design of each API, making sure they are consistently designed across teams, but there are a mix of dimensions that API governance looks that overlap with areas above. Here are some of the types of things being tested when it comes to API governance.

Specification - Is the specification provided for each API a valid representation of the spec.
Information - The naming, description, and details of an API, making sure they are sufficient.
Versioning - Making sure that each API is using the same approach to managing change.
Paths - Considering the overall design of available API paths available across all APIs.
Requests - Ensuring that the structure of API requests are following a common set of patterns.
Responses - Ensuring the the structure of API response are following a common set of patterns.
Schema - Making sure the common schema are applied across requests and responses.

I can get finer grain on requests, responses, and other aspects of governance, but this provides a general set of what I’ve seen when it comes to tangible API governance testing. The discipline of manual or automated API governance testing isn’t as mature as other areas of API testing, but this will prove as a nice base for considering API governance as part of the overall API testing suite. The big difference with these types of tests is that you aren’t testing the results of each API instance, you are testing the surface area of the API and comparing it against all the other APIs, and an organizational, or industry wide set of specifications and standards.

Considering How we Test API Operations

Now I want to continue pushing the notion of we are testing. I want to imagine where API testing needs to go to continue stabilizing how we deliver APIs across an organization or an industry. This is the dimension of API testing that isn’t about any single API, or about the design of all APIs, and more about the entire API operations. The overall quality and reliability of APIs aren’t just about the APIs themselves, and is often about how APIs are delivered across their overall lifecycle, as well as the resources available to operate and support each one. You can have an API that passes all of its instance testing and overall governance, but if other aspects of API operations are falling short, it doesn’t matter--here is what I am imagining testing looking like at the operational level.

Documentation - This is a duplicate from above, but I feel like belongs here and not in the list above.
Testing - Actually testing what testing is in place for each API, making sure APIs are consistently tested.
Monitoring - Having consistent approach to scheduling or automating testing across API operations.
Workspaces - Making sure all APIs possess a known workspace where everything about them can be found.
Discovery - Another one from above that I think belongs in the overall operational view of testing.
Management - Applying consistent authentication, plans, rate limiting, and reporting across all APIs.

There are other dimensions of API operations I could advocate testing for, but this reflects the most important dimensions of what is happening across API operations today. You see two of these areas already reflected in the types of testing that I cam across studying API testing above. Each of these areas of operations possess infrastructure that have their own APIs, which also provide machine readable artifacts that can be tested. Either adding a 3rd layer to this API testing conversation, or as I see it, just another set of API tests that should be run, but can be categorized into separate buckets. Historically I have seen all of these things as different aspects of API Operations, but now I am beginning to see them as just a suite of different tests being run, that test each individual API instances, the surface area of all APIs, or testing using the infrastructure behind API Operations.

Bringing a Little More Clarity to API Testing

Obviously all of this is going to take a lot more work before I get it into a more coherent state. I am not confident in my understanding of all of the API testing types listed above, and I am sure that there are other ways that I want to be able to articulate governance and operational level API testing in better detail. I also want to think about the common patterns that exist across all of the existing types of testing before I move to far down the road. I see some like business value and who is testing, but I want more time to dive into each individual area and do more research, before I make any assumptions. Really I think the big revelation of this post for me is where the artifact we are testing originates, which is from one of these three locations:

API Instance - Testing the request and response of a specific API instance.
API Surface - Testing the design and surface area of each individual API.
API Operations - Testing the operations that exist around all APIs.

I can envision different categories of tests existing across these three areas. I feel like next major dimensions of this conversation will be who is doing the testing, and the tangible business value associated with each test, or bundle of tests. I am also thinking that the delivery mechanisms for testing will heavily influence this conversation, providing a different set of realities if we are running tests manually, scheduling via a monitor or agent running in some cloud region, as part of a CI/CD pipeline, or some other event that gets triggered as part of API operations. I have a lot more research and thinking to do on this subject before I’ll reach the levels of clarity I am looking for here, but this post really helped me get things rolling. Being able to see each of these dimensions of what we are testing has set in motion a whole other set of thoughts for me, depending on whether we are testing the API instance, surface, or operations.

Keeping Api Entropy Low Is Needed To Continue Api Growth And Expansion

Sat, 30 Jan 2021 00:00:00 +0000

published: true

title: ‘Keeping API Entropy Low is Needed to Continue API Growth and Expansion’

image: https://s3.amazonaws.com/kinlane-productions2/algorotoscope-master/nazi-invasion-steam-engine-iceland.jpg

I love the word entropy. It means so many things to so many different people. In means different things in the physical realm versus the informational or virtual realms. It is one of those big words you can say to confuse people, or actually get yourself in trouble being able to actually defend your position on what entropy means when around smart people. It is said that famed mathematician John Von Neumann had told fellow mathematician and creator of Information Theory that he should use the word entropy instead of information because that no one understands entropy and then you can win arguments about your theory. I also read that Shannon and Norbert Weiner also dueled over the meaning, seeing it in very opposite ways. I like the word for all of these reasons, but mostly because of its relationship to uncertainty.

In the realm of physics entropy means, a thermodynamic quantity representing the unavailability of a system's thermal energy for conversion into mechanical work, often interpreted as the degree of disorder or randomness in the system In the more virtual realm of information theory, the entropy of a random variable is the average level of information, surprise, or uncertainty inherent in the variable's possible outcomes. While I could easily use entropy as a metaphor for APIs in the physical sense, it is really the appropriation of it for information theory that really brings it home for me. Entropy in Shannon’s time was centered around noise or static in communication, but when you look at how APIs enable communication between systems, and allowing web, mobile, device, and network applications to communicate, this static or noise reveals itself as friction, errors, outages, instability, unreliability, and misaligned business and political priorities.

Think about the level of information one encounters when it comes to putting APIs to work these days. Consider the surprise one encounters when seemingly similar API resources speak entirely different dialects or behave inconsistently. Stop to think about the amount of uncertainty that exists between each version of an API, let alone each version of an API across the increasing number of APIs your average organization depends upon today. The velocity at which an organization moves, or should be moving, considered alongside the velocity of each individual API being published or consumed, reveals the reality of API entropy on the ground within your average enterprise organization, as well as the industries in which they operate. When an API and resulting integration or application is brand new API entropy is usually relatively low, but as momentum increases, and the number of consumers increases, entropy will ultimately increase and contribute to the eventual decline of an API, the applications they power, and even platform that they operate on.

APIs work best where the cognitive load involved with putting them to work is low, where common patterns are employed, and predictability and reliability are high. The reduction of API entropy within API operations and communities can come in many forms. Popular API documentation tooling like Swagger UI, Slate, Postman, and Redoc help reduce API entropy by taming the information surrounding each API. API specifications like OpenAPI, AsyncAPI, and JSON Schema help reduce API entropy by standardizing the surface area of each API and reducing it to a common vocabulary that an increasing number of developers, commercial services, and open source tooling are familiar with. OpenAPI forces API providers to reduce the information a consumer needs to digest when integrating with an API, and has the potential to reduce surprise and uncertainty throughout the life of each API, and the journey of each consumer putting it to work.

When you also deliver APIs using common standards that are defined using open specifications like OpenAPI, AsyncAPI, and JSON Schema, you further reduce API entropy across operations and within an industry. PSD2 defined using OpenAPI and JSON Schema reduces API entropy for banks, and across the European (and UK) financial sector. The CFPB in the United States considers the adoption of similar rules here in the US will further reduce entropy for banks and the financial sector on this side of the pond, but ultimately globally. FHIR defined using OpenAPI and JSON Schema will reduce API entropy for healthcare providers, and the entire healthcare sector in the United States. Something being driven by agencies like CMS, HHS, and the Department of Veterans Affairs through regulatory guidance and adoption of FHIR, while also actively using OpenAPI as part of their API lifecycle. We are seeing this play out in the travel, transportation and other leading sectors that are looking to reduce the resources it takes to deliver web, mobile, device, and other infrastructure, while also ensuring that enterprise organizations are more agile, nimble, and competitive, and the industries they operate in are more healthy, active, and interoperable.

API entropy is experienced as friction throughout the API lifecycle. High API entropy within business sectors that power our economy can be seen in the lack of access and interoperability between vendors, institutions, and government agencies. When you are down in the weeds of an organization API entropy just looks like API chaos, and from the position of someone who studies the 250K of how APIs are evolving, you begin to see API entropy as something similar to what is happening to our physical natural environment(s), something that long ago reach unsustainable levels, but is something that can be managed if more of us realize that we are all in this together, and get to work on more specifications, standards, and guidance, while centering our commercial services and open source tooling around these standards. Keeping API entropy low is needed to continue API growth and expansion at the current velocity, and will be something that slows the growth in key industries and large sections of our economy if we don’t reduce the entropic pressure that exists across the API factory floor that first emerged in leading technology companies, but in the last decade has emerged within every mainstream company, organization, institution, and at all levels of government.

Examples Of Minimum Viable And Complete Landscape Apis Json Index Files

Fri, 29 Jan 2021 00:00:00 +0000

published: true

title: ‘Examples of Minimum Viable and Complete Landscape APIs.json Index Files’

image: https://s3.amazonaws.com/kinlane-productions2/algorotoscope-master/norman-rockwell-ruby-bridges-border-crossing-through-fence.jpg

<p>I am preparing for the next version of APIs.json and I am taking another pass over what the specification is, but also taking a fresh look at why the specification exists. Part of this fresh look involves assessing what I consider to be the low bar for an APIs.json index, as well as what the entire scope might look like. To help me (and you) understand the scope of what APIs.json or APIs.yaml can do when it comes to indexing our API operations and making them discoverable in a machine readable way.</p>

The most important thing to remember about APIs.json is that it is all about indexing API operations in a way that acknowledges that there are two sides of this API integration game which need to be indexed and made discoverable across an organization.

Human Readable - Providing URL references for the aspects of API Operations that human stakeholders will need to understand what is happening with APIs and how to put them to work in applications.
Machine Readable - Providing URL references for aspects of API Operations that are system and applications can use to make sense of API Operations, accessing as a machine readable definition.

APIs.json is about indexing all of these human and machine readable aspects of our API Operations with an emphasis on evolving human readable elements to also be machine readable so that as much of the surface area of our API operations is indexable by other systems and applications as possible.

There is another dimension of APIs.json as a specification that is helpful to understand beyond the human and machine readable elements, by providing a vocabulary for describing two separate scopes of API operations, helping define the scope of information provided.

API Specific Properties - References to details about a specific API, providing human and machine readable details about an API, so that consumers understand everything that is possible.
Common Properties - References to details that support all APIs being indexed, providing information about how to onboard with a platform, and understand what is possible across A?PIs.

Similar to migrating human readable elements to be more machine readable, I’d say the goal in this dimension is all about standardizing our API operations across all of the APIs being made available, providing common set of resources across operations, while providing only what is needed to define the surface area of each individual API—helping be consistent in the business and technology of our API operations.

An Example Minimum Viable APIs.json File

To demonstrate what is possible with APIs.json when it comes to indexing operations I want to have an example of what I’d consider to be the minimum viable example of what an API Operations might look like in the wild.

That represents what I’d like to see from all APIs, providing the minimal amount of details and operational resources that any API should possess, acknowledging that the API itself isn’t enough, and there will always be other supporting elements needed.

An Example Minimum Viable APIs.yaml File

Because there are many situations where having a YAML edition of the index for your API operations is preferred, here is an example of the same minimum viable document, but in a YAML format, allowing the index of APIs operations to remain machine readable, but also be a little more human readable at the same time.

This minimum viable example APIs.yaml provides the same baseline as the APIs.json move, providing a human and machine readable artifact that can act as a blueprint for how API providers can think about their operations, while then also providing an index that describes what is happening after the blueprint is followed.

An Example Landscape APIs.json File

Moving all the way to the other end of the spectrum, I wanted to demonstrate the full scope of APIs/.son or APIs.yaml, and the many different aspects of API operations I see across the existing API landscape. Providing a robust look at what an API index might contain when it comes to more mature API Operations.

It is unlikely any single API operations would have all of these characteristics, but it does inform the most common aspects of API operations that are already being applied, as well as some advancements and forward looking properties that point to where we should be going with our API operations.

An Example Landscape APIs.yaml File

Similar to the MVP APIs.yaml, I also want to provide an APIs.yaml example of the full landscape index, showing how the full scope of API operations can be defined as YAML, helping make it more accessible to human and more specifically non-developer muggle types of humans.

I have organized the specific API, as well as common API properties into logical groups, and named them so that they are self explanatory, helping articulate their purpose, so that I can assess the role they play in overall operations. Showing what all of the different properties of APIs operations might look like depending on the types of resources being made available via APIs.

Indexing the API Operational Spectrum

This set of artifacts helps me see the entire spectrum of API operations from not just a discovery perspective, but also as a checklist or blueprint for how API providers should be thinking about their operations. Like OpenAPI, AsyncAPI, and JSON Schema, APIs.json doesn’t just do API discovery, it helps you think more deeply about your API operations in a standardized way that will reflect other API operations, and is something that will be more familiar to API consumers. One of the reasons there is still so much chaos in the API space when it comes to API discovery, on-boarding, and other aspects of a speedy integration, is that we don’t have a common vocabulary for describing and communicating around our API pperations--Steve Wilmott and I intended on changing that.

I will use these two distinct APIs.json and APIs.yaml artifacts to help articulate what is possible with the specification, but also use to define the spectrum of the API landscape when it comes to how we should be publishing a suite of human and machine readable artifacts describing our operations, with a perpetual forward motion towards making our API operations more machine readable. I will use these artifacts to inform how I work to on-board new API [roviders to the concept of APIs.json, while also using it as a point of reference for where I’d like to be moving the conversation when it comes to API discovery across enterprise organizations as well as the industries they operate within. Leveraging APIs.json to index enterprise API operations, but then also provide the discoverability needed for the public and industry layers of our API operations.

Evaluating Apis Json Api Property Types Alongside Openapi Extensions

Sat, 23 Jan 2021 00:00:00 +0000

published: true

title: ‘Evaluating APIs.json API Property Types Alongside OpenAPI Extensions’

image: https://s3.amazonaws.com/kinlane-productions2/algorotoscope-master/birth-of-a-nation-chess-in-the-park-with-pigeons.jpg

I am giving some much needed love to my APIs.yaml and API.json API discovery format while using the work to also just think about the wider API landscape. This is the original intent behind APIs.json…to help me make sense of the API landscape. Most folks are thinking about APIs as a producer, or as a consumer—hopefully both. However, there are fewer people who also pay attention to the entire API landscape, and APIs.json emerged out of this need for me in 2014. In 2020 I find myself totally immersed in the API landscape as part of my API Specification Toolbox project which grew out of Postman becoming a member of the OpenAPI Initiative, and I wanted toht spend why Saturday afternoon (I know I have a problem) thinking about the API landscape at the 250K.

APIs.json API Property Types

Using APIs.json or APIs.yaml you can define the properties of one or many APIs for a variety of purposes. Each API you describe can have an array of properties which can be common human readable elements like documentation or sign up, as well as machine readable properties like OpenAPI, AsyncAPI, or JSON Schema. APIs.json property types are intentionally human or machine readable with the intent on evolving every human readable element to have a machine readable element—think the relationship between documentation (Human) and OpenAPI (Machine) when understand the objective.

I have several thousand API Providers and API service providers in my API Evangelist database. I track on the properties of all of these companies, organizations, institutions, and government agencies that I come across. When I group the property types across the thousands of APIs I have indexed, I end up with this default set of building blocks for API providers, grouped by different areas of the API lifecycle.

Onboarding

These are the properties of any API operation that help consumers, or any other party onboard with the concept of an API, begin using the API, as well as allowing them to return on a regular basis to get what what they need for integration.

About - The overview of what an API or provider offers to consumers.
Getting Started - How you get started with actually consuming an API.
Authentication - What is involved with authenticating with an API.
Documentations - The human readable documentation for APIs.
Plans - What the pricing, access, and rate limits are for each API.
Signup - Where you sign up to actually begin using an API.
Login - How you login back into the platform for each API.
Dashboard - The landing page for consumer after logging in.

These API property types are essential to helping API consumers understand what is possible and assisting them in getting up and running with each API. As the number of APIs each of us depends on increases, friction at any of these areas might be the deciding factor for whether a consumer successfully onboards with an API or not.

Definitions

These are the common machine readable API specifications in use by API Providers, offering a suite of ways for API providers to define and communicate what their APIs actually deliver. Providing a common vocabulary for describing the surface area of our APIs in a way that can be used across every stop along the API life cycle. Here are the leading specifications that are defining each individual public, partner, or private API being delivered.

OpenAPI - The leading specification for describing HTTP 1.1 request and response, or REST APIs.
AsyncAPI - The leading specification for describing event and message driven multi protocol APIs.
JSON Schema - The default standard for how you define and validate each schema being used.
Postman Collections - A dominate format for executing and realizing what is possible with each individual API.

As an industry we are doing well at providing machine readable definitions for each individual API instance, describing the surface area of each API, the payloads, and the technical details of integration and consumption. However, as I’ll explore later, we are don’t such a good job at ensuring all the building blocks of our API operations are machine as well as human readable—something APIs.json is looking to change.

Developers

When it comes to the more developer centered, and more technical aspects of APIs operations I see API providers offering, there are a handful of ways they are incentivizing developers to put their API resources to work. These are the most common technical building blocks of API operations I have captured over the last ten years of mapping out the API landscape.

SDKs - Providing a complete list of SDKs available in every available programming language or popular platform.
CLI - Offering up command line interface tooling for developers to use as part of integration and the CI/CD process.
Open Source - Having a dedicated page to showcasing the open source solutions that are available for developers.
Github - Embracing the social coding platform Github when it comes to managing code and the relationship with developers.
Applications - Providing a directory or showcase of applications that are built on top of an API, highlight what has been built.
Integrations - Offering up a ready to go suite of integrations with other APIs for consumers to put to use in a simple way.
Webhooks - Allowing web hooks to be triggered upon relevant events that occur across a platform, making APIs a two-way street.

These are the meat on the bones of successful API integrations. This is what developers and end uses depend on to successful put a platform to work. Currently APIs.json is just linking to each of these pages, providing a URL for a human to follow. You can describe web hooks using OpenAPI or AsyncAPI, but there will need to be a lot more evolution from human to machine readable in coming years if we are going to keeping scaling this Frankenstein of an API ecosystem we’ve built.

Change

Change is inevitable in the API universe, and those who are better at communicating around change are almost always the ones who are better at managing it, while reducing the negative impact of change on API consumers. There are the handful of ways in which I see API Providers actively communicating about the change that is happening across their API-driven platforms.

Versioning - Providing all the information needed to understand the semantic, or other versioning applied to an API.
Road Map - Being more organized and communicative about defining where an API is headed, as far into the future as possible.
Change Log - Ensuring that everything that has changed with an API is documented, leaving a trail of everything that changes.

This is one area that will also have to get more machine readable if we are going to scale APIs to meet the demands of the global economy. We will have to learn to to programmatically navigate change if we are going to further automate integration. Applications will need to be equipped to make their own decision about which versions of an API they use. Ideally, there would be hypermedia baked into our API design to help mitigate against change, but alas that isn’t always present. ;-)

Communications

Being able to communicate with consumers is one of the most vital characteristics of an API provider. If you aren’t able to provide a regular stream of communication around of the value offered by an API, and able to keep consumers informed, you probably won’t be very successful in operating an API. While these channels may shift when it comes to partner or internal APIs, the need for communication remains the same—here are the most common public API communication property types.

Newsletter - Providing a regular email newsletter about what is happening with an API.
Blog - Having an active blog with a feed for consumers to tune into around an API.
Twitter - Providing an active presence on Twitter, being social with the API.
LinkedIn - Having a professional presence for an API using LinkedIn.
Youtube - Publishing or aggregating videos around an API on Youtube.
Press - Providing an inventory of press regarding the business value of an API.

What I like about this section is that every one of these API property types have APIs, except usually press. Blog have RSS and Atom, and Twitter, LInkedIn, and Youtube all have APIs. I have a wealth of other communication property types that I track on, but these reflect the most dominant channels that API Providers are using to deliver one side of the API feedback loop that is essential to an APIs success.

Support

The other half of the API feedback loop is about ensuring you are adequately supporting your community and API consumers. There are a wealth of common as well as creative ways API Providers support their communities, but these are the top ways I am tracking on when I profile and index each API I track on.

FAQ - A well defined and organize list of common questions and their appropriate answers.
Email - Simply providing an email address for consumers to use when looking to get support.
Contact Form - A URL to the contact form to use for submitting support requests for an API.
Tickets - Allowing consumers to submit a support ticket and get help with API integration.
Issues - Using Github or other issue format to allow the public or private submission of issues.
Stack Overflow - Leveraging the popular Stack Overflow platform, outsourcing your support.
Forums - Operating your own forum to support the community around an API.
Status - Maintaining a 24/7 status of the uptime and availability of all APIs over time.

These support centered property types complete the feedback loop for API Providers, processing everything “heard” from support and including it as part of the API road map, while then using it to informing the communication side of the feedback loop, establishing a virtuous cycle between provider and consumer.

Resources

You can always tell when an API provider has invested in their platform and community because there are always a wealth of resources available to support the community. These are the leading resources you will find in a modern API provider’s community, further adding to the information available to put a platform to work.

Case Studies - Simple overviews of how a platform has been put to work by another consumer.
Tutorials - Easy tutorials that walk consumers through different aspects of how an API works.
Webinars - Live session educating consumers about what an APIs, with videos posted afterward.
Streams - Providing unscripted problem solving live streams that help demonstrate the value of an API.
Videos - Offering up videos on a variety of relevant aspects of what an API will do for consumers.
Podcasts - Publishing or participating in audio podcasts that help relay what an API does for consumers.

Take a look at Twilio if you want to see how to do API resources. All of these resources help educate consumers about what is possible, and help not just reduce friction when integrating with an API, but also helps when it comes to expanding the consumption of API resources by existing applications.

Legal

Now for the boring, but essential legal layer of API operations. These API property types have a profound effect on what is possible with an API, and while we have the beginnings of investment in the legal side of API operations, we have a lot of work to help make all of these API properties something a machine can understand.

Terms of Service - The legal terms of use / service that guide all usage of a platform.
Privacy Policy - How the privacy of developers and consumers are defining for a platform.
Licensing - The licensing that exists for data, backend code, the API definition, and client code.
Service Level Agreement - What the agreed upon SLA that exists between provider and consumer.
Security Policy - Being upfront regarding what the security practices and policies for an API are.

This is the area that is furthest out of view for most developers, and it is something that will continue to define the relationship between end-user, developers, and the provider. We are seeing significant contributions to the machine readability of licensing, SLA, and security side of API operations, but we need a lot more work when it comes to making sure the terms of service and privacy policies are something that can be processed programmatically.

I track on a couple hundred property types for API providers, but this reflects the cream off the top of the essential API operational characteristics. One of the most misunderstood aspects of APIs.json is that most people think it is for describing APIs, but it is really about describing API operations, providing a snapshot of the technology, business, and the politics of APIs. This list is defined by ten years of crawling, harvesting, and profiling APIs, studying what the healthy and not so healthy aspects of their API operations are. APIs.json has always been about mapping out this aspect of individual API operations, but more importantly entire business sectors, and the overall global API landscape—something I will be kicking into high gear by doing even more research on how the landscape is currently being mapped out using OpenAPI, AsyncAPI, and JSON Schema, while being realized at throughout the API lifecycle with Postman collections.

Types of API Extensions

The list of API property types for APIs.json has always been defined by how API providers are doing what they do to support their communities. Overlapping with this work, but emerging from a variety of OpenAPI focused projects I am working on, and stemming from me assuming my role as co-chair of the business governance board (BGB) for the OpenAPI Initiative (OAI), I find myself further studying the ways in which the OpenAPI specification is currently being extended. After looking through the OpenAPI extensions from 14 different providers I was able to break down a short (ha!) list of the ways in which API Providers are extending the specification to meet their needs for each API, but also their needs across the entire API lifecycle. Here is the breakdown of OpenAPI extensions I found while doing the work to add them to the API Specification Toolbox.

Documentation - Expanding on the spec for publishing documentation.
Examples - Enriching the examples for each API for use across the API lifecycle.
Parameters - Providing more details about each of the path, header, or question parameters.
Parameter Groups - Helping group parameters to help make them more organized.
Defaults - Offering up a set of default values for different layers of an OpenAPI.
Enumerators - Adding more information about enumeration for properties and parameters.
Versioning - Allowing there to be more details about the versioning of each API or schema.
Role Based Access Control (RBAC) - Adding a layer for RBAC across an API and their surface area.
TLS - Providing details about the transport level security being applied to each API.
Compression - Outlining the type of compression that is available as part of each API.
Validation - Describing the validation that occurs when an API is consumed or tested.
Authentication - Expanding on the overall authentication and access management.
Basic Auth - Providing more detail about Basic Auth that OpenAPI doesn’t provide.
API Key - Providing more detail about an API key that OpenAPI doesn’t provide.
OAuth - Providing more detail about oAuth that OpenAPI doesn’t provide.
JWT - Providing more detail about JWT that OpenAPI doesn’t provide.
Environments - Adding additional details about development, production or other environment.
Management - Defining the management layer that is in place for each API.
Policy - Allowing policies to be applied to each API being defined by OpenAPI
Rate Limit - Defining what levels of consumption are available for each API
Throttling Tier - Further defining the overall access or throttling tiers available.
Configuration - Providing configuration for how an API should be operating.
Gateway Request - Additional information regarding the request surface of an API.
Gateway Response - Additional information regarding the response surface of an API.
Integrations - Detailed information about the integration with another API behind an API.
Backend - Further details about backend integrations and connections behind each API.
Dependencies - Machine readability details about all of the dependencies an API has.
Languages - Machine readable information about the language available for an API.
CORS - Details about how cross origin resource sharing is addressed for an API.
Code Samples - Providing samples or snippets for all available API paths.
Client Code Generation - Information regarding what client side code generation.
Server Code Generation - Information regarding what server side code generation.
Requests Scripts - Defining scripts that are executed with each API request made.
Responses Scripts - Defining scripts that are executed after each API response.
Translation - Details about what translation occurs as part of each API request or response.
Filtering - Information regarding what filtering is available or applied to each API request or response.
Aliases - Providing details regarding aliases for different elements of each API.
Pagination - Offering more structure regarding the pagination of API resources.
Triggers - Defining what triggers are available or executed as part of API operations.
Notifications - Outlining available notifications that are made when an API is executed.
URL Encoding - Providing more details about how a URL is encoded as part of each API resource.
Data Store - Associating a data store with each API being defined as an OpenAPI.
Caching - Adding information regarding how each API is being cached as part of operations.
Security - Further expanding on how security is handled when it comes to each API operations.
Import - Providing additional information that needs to be considered or applied upon OpenAPI import.
Export - Providing additional information that needs to be considered or applied upon OpenAPI export.
Audiences - Offering more detail about the audience an API is intended for as part of each OpenAPI.
Capabilities - Having a machine readable defining of what each API is capable of doing.
Traits - Further outlining the traits of each of the APIs, helping further flesh out what each one does.
Tags - Adding more rich detail beyond the base tagging provided by OpenAPI.
Tag Groups - Helping group tags, adding another layer of organization to each API.

I tried to organize all of these OpenAPI extensions into groups and failed, but I did manage to generalize them a bit. Some of the extensions were very specific, while others were very broad. It is definitely like looking at apples and oranges, but it does provides a rich look at how API providers are bending the OpenAPI specification to meet their operational needs. It is something that should definitely inform the OpenAPI road map, but for me, from the operational vantage point of APIs.json, I have a lot of questions about this list regarding where each extension should actually exist. Asking if each extension be part of each individual OpenAPI, or it should it be overlaid, or maybe just exist as part of some other looser coupling. I won’t be able to answer these questions until I think more deeply about how each of these extensions apply to each individual API as well as across the lifecycle of the API. This list of extension is very OpenAPI centric, and I’d like to put some more thought into whether this extension should also be applied in AsyncAPI, JSON Schema, or maybe a Postman collection…or maybe D) all the above, or E) none of the above.

An API.json Index

The easiest way to understand why we created the API.json specification is to think of each definition as an index of a publicly available API. A single machine readable index for Twilio, Twitter, or more revealing—Slack. Something that each API provider creates to define their API Operations, or something that is created by someone like me to index a single API, and entire industry or universe of APIs. However, when you when you start looking at all of the API extensions above you see elements that may also be defined by an API consumer, or for the benefit of an API consumer, and not the API provider. Pushing the purpose of an APIs.json and leveraging it as something that benefits both API provider and/or consumer. However, after profiling thousands of API using APIs.json I began to also see that a more robust and mature APIs.json for an API operations can also become a sort of blueprint, and something that can be used as a checklist for delivering and operating an API.

An APIs.json Blueprint

After you see more robust APIs.json indexes like from Twilio or Stripe you begin to see that they can become a blueprint for how to deliver APIs. If you just swap out the OpenAPI and Postman collections for each of the Twilio and Stripe APIs and add your own, you realize that have a blueprint that you can follow when operating your own API. This is all I have done with the APIs.json property types listed out above—I have defined APIs.json for thousands of APIs, then providing a distinct and grouped set of the most common property types applied by API providers. Adding the list of the API extensions to this list helps us further understand how API providers are not just operating their APIs on the ground on their API factory floors, it also reveals how they are trying to standardize how they do what they do, and more effectively communicate it using the OpenAPI specification.

An APIs.json Orchestration

Beyond defining existing APIs, as well as delivering new APIs that are consistent and possess all the properties needed for success, APIs.json can be used to define a suite of different APIs that are used as part of a larger orchestration or delivering web, mobile, device, and network applications. If I am defining a specific type of integration that utilizes a mix of different APIs, orchestrating across APIs one time or over a longer period of time via a scheduled or CI/CD actions, or delivering a specific type of application, it will benefit me to have everything articulated as a single APIs.json definition. Further pushing how not just APIs.json or APIs.yaml works, but demonstrate how multiple machine readable specifications like OpenAPI, AsyncAPI, JSON Schema, and Postman collections can be used to orchestrate operations. Pushing all of these API specifications to work more like Terraform or Ansible, but for specifically defining the API landscape and API operations.

Seeing the APIs Landscape Using APIs.json

Assimilating all of the details I outlined above into a single coherent index, blueprint, orchestration, or other APIs.json definition will take a lot of work. However, I would like to take what I’ve learned with this round of assessment of APIs.json property types, and research into current state of OpenAPI extensions and establish a base APIs.json definition that I could use as a lens for me to look at the both the world of APIs, but also the API specifications that are used to define that world. This APIs.json, or more specifically APIs.yaml file isn’t meant to be a real world index, blueprint, orchestration or other type of definition, it is just meant to be a sort of API landscape toolbox for me to use as I make sense of the space today, and where it might be going.

This isn’t meant to describe a single API. It is unlikely any single API would or should possess all of these individual or common properties. This is just meant to be a snapshot what is possible when it comes to operating, consuming, and orchestrating with one or many APIs. It provides me with the scaffolding to process and lay each of the extensions I discovered above, helping me evaluate whether each of the individual extensions should indeed be augmenting the OpenAPI, or maybe the AsyncAPI, JSON Schema, or possibly stand on its own legs as an independent specification or overlay. I am looking to establish a more formal way for me to process extensions that I find across the API sector, as well as better understanding how services and tooling are using API specifications to do what they do.

I am not pulling all of this out of thin air.It is based upon mapping out the API landscape for over ten years--recording what I see. The OpenAPI extensions, and continued research into how API providers and service providers are extending the specification also reflects the realities on the ground across API operations. Additionally when you take into account that the infrastructure we use to design deploy, and manage APIs across the API lifecycle also have APIs, you begin to see how much of the machine readable meta data we need to fill out this APIs.json already exists, and all we need is a place to hang it. This isn’t an artifact that you will be crafting by hand. It is one that we will have to rely on API service and tooling providers across the spectrum to output and publish for us.

Ok, I will pause there. Next I will begin to hang all of these extensions on this APIs.json scaffolding I have developed. I will need to add the bones of each of the existing machine readable APIs.json property types like OpenAPI, AsyncAPI, JSON Schema, and Postman collections. Then work my way through the OpenAPI extensions and evaluate whether they indeed should be hung within the OpenAPI, or possibly somewhere else. This is what I mean by being able to see the API landscape using APIs.json. It will help me more coherently be able to apply each individual extension, but instead of being siloed within the context of the OpenAPI namespace, I will be able to consider each of the extensions in a multi-protocol way, while also thinking about each stop along the API lifecycle in a single or multi-protocol way. Helping me better understand the API landscape, but also the current API specification landscape, and make more educated decisions around where I or we invest. Providing me with a more structured and disciplined approach to contributing to the API present and future.

What Does Open Mean In The World Of Apis

Mon, 11 Jan 2021 00:00:00 +0000

published: true

title: ‘What Does Open Mean in the World of APIs?’

image: https://s3.amazonaws.com/kinlane-productions2/algorotoscope-master/america-immigration_dumping-ground-do-not-enter-sign.jpg

The word “open” gets thrown around so much in the API space I find myself needing to regularly ground myself in what it actually means. It gets thrown around in so many different ways that I find eventually I start to believe the bullshit and regularly have to pause and recalibrate. It is one of those words that has been captured and used by people looking to manipulate the space so often it often means the exact opposite of what we all think it means, but it is also such an important word that I think we have to fight to keep it in our vocabulary. I feel like I am perpetually fighting with some invisible force over one of the front doors to the house we all live in, and I am determined for the door to not get completely shut, and I have to make sure the door actually ends up go where someone expects when they do open the door.

Let’s begin by actually taking inventory regarding the many different ways people use the word “open” to describe the world of APIs. Gathering all the ways it gets wielded in service of publishing and consuming APIs. Some of the ways in which it gets wielded are good, and some of them are bad. At this point I am just looking to be able to see the spectrum of use, and I am not looking to entirely pass judgement on anyone who uses it in specific ways. Here is the laundry list of ways “open” is wielded that impact the landscape I pay attention to on a daily basis, and have been helping shape for the last decade.

Publicly Available - Open means something is available to anyone in the public, allowing them to use an API, and any resources around them.
Access - An API or supporting resource is open for access by the public or the developer community in general, making it something they can use.
Business - Stating that a digital resource is open for business, and consumers can purchase or be purchased in an ongoing way using an API.
Source - Focusing on the code behind or in front of an API, and acknowledging that APIs are built upon and evolved using open source code.
Specifications - Applying to API specifications like OpenAPI or OpenID, and ensuring that the specs we leverage across our APIs are always open.
Standards - Looking at specific industries and business sectors, and ensuring that the standards that are in place for industries are available openly.
Licensing - The word open is regularly applied to the legal side of the API world, which leverages the lawyers to loose up or tighten down the open screws.
Exploitation - Many APIs and the platforms they reside upon are open for exploitation, or possibly open to the exploitation of API consumers and end-users.
Discovery - Can I find the API? Is it easy to discover and explore for any potential consumer, and able to find via common, well-known channels.
Portal - Is the landing page or portal for the API accessible to the public, partners, or a private audience allowing them to learn about the API.

Documentation - Is there complete, up to date API documentation for the API that is accessible, providing an honest and open view of what an API does..
Paths - Do developers have access to all of the paths that are available for the API made available, or are there hidden paths available behind applications.
Enumerators - As a developer do I have access to all enumerated values that can be used for an API, and is it clear for me the values an API will accept.
Read / Write - Is an API just GET access, or can developers POST, PUT, and DELETE resources, ensuring the surface area of each digital resources is open.
Media Types - Are there standardized media types in use for each API response from an API, or are they cryptic, closed, and unique formats that aren’t open.
Sign-Up - Is the signup for the API self-service and allows developers to easily access and complete, and something that is open for all types of consumers.
Sign-Up Scope - How much information is required from developers to be able to get access to an API, or will I need to share my entire life story to get access.
Sign-Up Approval - Do developers have to wait for access and be approved before they can obtain keys to access, making open very subjective in reality.
Authentication - What is involved with authentication, and is the technical bar set too high for some developers, compared with the digital resources.
Design - Is an API intuitive, easy to understand, and accessible to everyone using it, or is it cryptic and hard to learn, and only understandable by a few.
Consistency - Are API patterns consistent across all APIs, alleviating extra work to put any of the APIs to work, or is each part some new to have to learn.
Rate Limits - What limitations are placed upon developers access to an API, limiting the usage of any individual API, putting full usage out of reach.
Rate Ceilings - Are there not just limits, but entire ceilings I cannot overcome at any point in development, rendering APIs not open to growth and scale.
Sandbox - Can developers access all APIs in a sandbox, so you do not have fear of only working with live data, restricting openness due to only a production state.
Status Codes - Are standard HTTP status codes returned allowing for understanding what each response means, leveraging the shared open meaning of the web.
Error Messages - Are there standardized and coherent error messages associated with each API, making things understandable, making failure more open.
Availability - Is there large periods of time when the API is not available for use within applications, leaving a platform only open during specific times.
Reliability - Is the availability of the APIs reliable, making it something developers can count on and put to use, and open in a reliable way across consumers.
Performance - Is the API performant and something you can actually receive responses in an acceptable time frame, and open to operating at speed we need.
API License - Is the interface for an API openly licensed so I can confidently integrate and use in my code, ensuring the surface area of the API is truly open.
Data License - Is data returned as part of API responses licensed in an open way allowing developers to use, keeping the data exhaust involved as open as possible.
Code License.- Are the cod made available as part of an API openly license for wide usage within applications, keeping backends and frontends as open as possible.
Tooling - Developers need to have access to the open source tooling they need to get their immediate job done, and meeting their day to day needs for this to work.
Usage Costs - Are the costs associated with usage of an API within reasonable ranges for developers, leaving it open to smaller players as well as larger ones.
Cost Increases - Are there regular cost increases that might move an API out of range of access, pushing the boundary for what is open out of range for some.
Support - Can I get access to support when it comes to accessing and using an API, and is a platform open to supporting it’s consumers at the levels needed.
Communication - Is there anyone home behind an API helping communicate about what is happening, providing a regular open heartbeat for consumers to hear.
Feedback - Are API operators open to feedback, ensuring that consumer feedback is a regular part of API operations, leaving the road map open to feedback.
Road Map - Is there any visibility into what the future will hold when it comes to the development road map, providing more openness to what the future holds.
Ideas - Does a platform foster the open creation, development, and cross pollination of ideas across the platform between API publisher and consumer.
Censorship - Is data or content actively being filtered or censored as part of API operations, and open are the filters that are in place on API infrastructure.
Privacy - Is there a privacy policy in place that protects developers and their usage of an API, regulating how open individuals lives are when on a platform.
Terms of Service - Is there a TOS in place that governs the API, and what are the constraints in place, and how open is the language used for terms of service.

Service Level Agreement - Are there any guarantees In place that govern the availability and costs associated with an API, being open about exceptions.
Regulations - The constraints of regulations will define how open or closed a specific business sector will be, setting the tone for open across APIs in that space.
Auditors - The transparency, openness, and observability of regulatory auditors will define each business sector being defined by the coming API regulations.
Investment - The accessibility of investment will continue to define the API space, opening up the building of the next generation of API instructor and applications.

This reflects the “open” knobs and dials that get used to control the open or closed nature of APIs we build and use each day. Open is a dance between provider and consumer and end-user when it comes to API, but also across providers and consumers within a specific sector or industry. In order for “open” to have meaning or not is about the trust that exists (or doesn’t) within a space. How a platform dials things in along these lines will define how open or closed they are, and this playing out across all the players within a sector will determine how open or closed the entire industry is. Each of these knobs and dials will reduce or introduce friction across the API factory floor for an enterprise organization, as well as within the industries they operate within. It represents where we can individually make a difference, as well as collectively, to put the open in not just APIs, but also across every part of our world that is being touched by technology—which is pretty much everything.

Ironically “open” is not a boolean in the world of APIs. In my experience the APIs that declare themselves “open” usually aren’t. However, I don’t want this to bias me, and I want to still believe that when folks use the word they have the best intentions. I want people to feel like they can use it without being immediately suspect. However, I want people to understand that the usage of this word comes with responsibility. That you will have to prove yourself. That there is a certain amount of accountability required here. This includes myself. I like to use the word open to describe a significant portion of what I do in the API space. I am an open storyteller. I am an open evangelist. 75% of the APIs I develop I would call open. I believe in the importance of open in the API space. It is why all of this works. However, I also understand the importance of doing some things in a proprietary or closed way, as long as it is done deliberately and for the right reasons. I believe in the role that open APIs plays across all of these levels, but I also understand that open is just a reflection of what is closed. I don’t see open as good nor bad. I see open as a meaningful way in which we can leverage the web to accomplish our individual and collective objectives and missions. I also am very skeptical of anytime open gets wielded, and will keep working to define what it means and doesn't mean as part of the API economy.

Taking A Fresh Look At Apis Across All The United States Federal Agencies

Sun, 27 Dec 2020 00:00:00 +0000

published: true

title: ‘Taking a Fresh Look at APIs Across All the United States Federal Agencies’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/america-under-socialism-capital_36728420065_o.jpg

It has been a while since I looked at the 250K view of what is going on with APIs across federal government agencies in the United States. Since working for the Obama administration in 2013 I am perpetually on a quest to map out what is happening across federal agencies, helping drive the conversation forward. I belieive APIs can make the most impact when our federal government helps lead the way, and I am looking to help push things forward in the following ways.

Keep mapping out the Federal Government API Landscape - I am determined to produce a map of the APIs that exist across federal agencies and keep the list alive and active.
Establish Public API Workspaces for Federal Agencies - I am looking to establish public API workspaces for agencies who are implementing APIs--helping do some work from the outside-in.
Refresh My Memory of What is Happening - I thrive on knowing what is going on across federal agencies, and doing these reviews pushes me to refresh my awarenss of APIs at this level.
Fire Up new Conversations - These sotries always rise up in the SEO game and bring in new conversations with folks who are doing interesting things with APIs in government.

I always learn a lot looking through the different government agencies. I learn even more wading through the different datasets, databases, and various incarnations of APIs. There is way too much work here than one person can handle, and much of what I come across labeled as an API really isn't an API, but could be with a little work. While doing these roundups I always reach a point where I feel like I am not doing enough, but ultimately I have to strike a balance between being comprehensive and just scratching the surface. Providing just enough information to allow me to plant seeds that might grow into new conversations down the road, while not spend all of my days sifting through the backwaters of federal government websites. This is my latest attempt to map out what is happenign with APIs across federal agencies in 2020, with some learnings and ongoing thought about what else can be done down below.

Administration for Children and Families (ACF)

The Administration for Children and Families is a division of the United States Department of Health and Human Services. It is headed by the Commissioner and Deputy Commissioner. It has a $49 billion budget for 60 programs that target children, youth and families.

Links

APIs

Unaccompanied Children Reporting API

Agency for Healthcare Research and Quality (AHRQ)

The Agency for Healthcare Research and Quality is one of twelve agencies within the United States Department of Health and Human Services. The agency is headquartered in North Bethesda, Maryland, a suburb of Washington, D.C.

Links

Website
Data

APIs

Agricultural Marketing Service (AMS)

The Agricultural Marketing Service is an agency within the United States Department of Agriculture, and has programs in five commodity areas: cotton and tobacco; dairy; fruit and vegetable; livestock and seed; and poultry.

Links

Website
Data

APIs

Agricultural Research Service (ARS)

The Agricultural Research Service is the principal in-house research agency of the United States Department of Agriculture. ARS is one of four agencies in USDA's Research, Education and Economics mission area.

Links

Website
Data

APIs

Alcohol and Tobacco Tax and Trade Bureau (TTB)

The Alcohol and Tobacco Tax and Trade Bureau, statutorily named the Tax and Trade Bureau and frequently shortened to TTB, is a bureau of the United States Department of the Treasury, which regulates and collects taxes on trade and imports of alcohol, tobacco, and firearms within the United States.

Links

Animal and Plant Health Inspection Service (APHIS)

The Animal and Plant Health Inspection Service is an agency of the United States Department of Agriculture based in Riverdale, Maryland responsible for protecting animal health, animal welfare, and plant health.

Links

Website

APIs

Animal Identification Management System

Bureau of Alcohol, Tobacco, Firearms, and Explosives

The Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF, BATF, and BATFE) is a federal law enforcement organization within the United States Department of Justice.[4] Its responsibilities include the investigation and prevention of federal offenses involving the unlawful use, manufacture and possession of firearms and explosives; acts of arson and bombings; and illegal trafficking of alcohol and tobacco products. The ATF also regulates via licensing the sale, possession, and transportation of firearms, ammunition, and explosives in interstate commerce. Many of ATF's activities are carried out in conjunction with task forces made up of state and local law enforcement officers, such as Project Safe Neighborhoods. ATF operates a unique fire research laboratory in Beltsville, Maryland, where full-scale mock-ups of criminal arson can be reconstructed.

Links

Bureau of Industry and Security (BIS)

The Bureau of Industry and Security is an agency of the United States Department of Commerce that deals with issues involving national security and high technology. A principal goal for the bureau is helping stop the proliferation of weapons of mass destruction, while furthering the growth of United States exports.

Links

Website
Data

Bureau of International Labor Affairs (ILAB)

The Bureau of International Labor Affairs is an operating unit of the United States Department of Labor which manages the department's international responsibilities.

Links

Website

APIs

Sweat & Toil

Bureau of Labor Statistics (BLS)

The Bureau of Labor Statistics (BLS) is a unit of the United States Department of Labor. It is the principal fact-finding agency for the U.S. government in the broad field of labor economics and statistics and serves as a principal agency of the U.S. Federal Statistical System. The BLS is a governmental statistical agency that collects, processes, analyzes, and disseminates essential statistical data to the American public, the U.S. Congress, other Federal agencies, State and local governments, business, and labor representatives. The BLS also serves as a statistical resource to the Department of Labor, and conducts research into how much families need to earn to be able to enjoy a decent standard of living.

Links

APIs

BLS Public Data API Signatures

Bureau of Ocean Energy Management (BOEM)

The Bureau of Ocean Energy Management is an agency within the United States Department of the Interior, established in 2010 by Secretarial Order.

Links

Website
Data

Bureau of Ocean Energy Management (BOEM)

Welcome to the Data Center. Here, users can access public information and data pertaining to the appropriate subject matter. Data are available via online queries, as well as downloadable PDF reports, ASCII files, and scanned documents available in PDF format. Some files are available for purchase on CD/DVD/Blu-Ray media. Information may be cross referenced among different subjects.

Links

Website
Data

Bureau of Reclamation

The United States Bureau of Reclamation (USBR), and formerly the United States Reclamation Service, is a federal agency under the U.S. Department of the Interior, which oversees water resource management, specifically as it applies to the oversight and operation of the diversion, delivery, and storage projects that it has built throughout the western United States for irrigation, water supply, and attendant hydroelectric power generation.

Links

APIs

Centers for Disease Control and Prevention (CDC)

The Centers for Disease Control and Prevention is a national public health institute in the United States. It is a United States federal agency, under the Department of Health and Human Services, and is headquartered in Atlanta, Georgia.

Links

APIs

Centers for Medicare and Medicaid Services

Visit Public Workspace

The Centers for Medicare & Medicaid Services (CMS), previously known as the Health Care Financing Administration (HCFA), is a federal agency within the United States Department of Health and Human Services (DHHS) that administers the Medicare program and works in partnership with state governments to administer Medicaid, the State Children's Health Insurance Program (SCHIP), and health insurance portability standards. In addition to these programs, CMS has other responsibilities, including the administrative simplification standards from the Health Insurance Portability and Accountability Act of 1996 (HIPAA), quality standards in long-term care facilities (more commonly referred to as nursing homes) through its survey and certification process, clinical laboratory quality standards under the Clinical Laboratory Improvement Amendments, and oversight of HealthCare.gov.

Links

APIs

Code.gov

Code.gov leverages the power of code sharing and collaboration to help the US Government cut down on duplicative software development and save millions of taxpayer dollars for the American people.

Links

APIs

Code.gov

Consumer Financial Protection Bureau (CFPB)

Visit Public Workspace

The Consumer Financial Protection Bureau (CFPB) is an independent agency of the United States government responsible for consumer protection in the financial sector. Its jurisdiction includes banks, credit unions, securities firms, payday lenders, mortgage-servicing operations, foreclosure relief services, debt collectors and other financial companies operating in the United States.

Links

APIs

Consumer Product Safety Commission (CPSC)

The United States Consumer Product Safety Commission is an independent agency of the United States government. The CPSC seeks to promote the safety of consumer products by addressing “unreasonable risks” of injury; developing uniform safety standards; and conducting research into product-related illness and injury.

Links

APIs

Consumer product Safety Commission (CPSC) Recalls

Corporation for National and Community Service (CNCS)

The Corporation for National and Community Service is an independent agency of the United States government that engages more than five million Americans in service through AmeriCorps, Senior Corps, the Volunteer Generation Fund, and other national service initiatives.

Links

APIs

Volunteering and Civic Life in America

Department of Agriculture

USDA is committed to providing safe and nutritious food, conservation, sustainable food production and revitalizing rural America.

Links

APIs

Department of Commerce

Commerce.gov is the official website of the United States Department of Commerce and Secretary of Commerce.

Links

APIs

Department of Commerce

Department of Defense

Established in 1949, the U.S. Department of Defense is America's oldest and largest government agency. Secretary of Defense Ash Carter currently heads the organization which employs over 1.4 million active duty service members and 1.1 million National Guard.

Links

Department of Education

Welcome to my page. If you're looking for the official source of information about the U.S. Department of Education, please visit our homepage at www.ed.gov.

Links

APIs

Department of Energy

Building the new clean energy economy. Reducing nuclear dangers & environmental risks. Expanding the frontiers of knowledge via scientific research.

Links

Department of Health and Human Services (HHS)

The United States Department of Health and Human Services (HHS), also known as the Health Department, is a cabinet-level department of the U.S. federal government with the goal of protecting the health of all Americans and providing essential human services. Its motto is "Improving the health, safety, and well-being of America".[3] Before the separate federal Department of Education was created in 1979, it was called the Department of Health, Education, and Welfare (HEW).

Links

APIs

Health Human Services Media

Department of Homeland Security

The U.S. Department of Homeland Security (DHS) is a cabinet department of the United States federal government, created in response to the September 11 attacks, and with the primary responsibilities of protecting the United States and its territories (including protectorates) from and responding to terrorist attacks, man-made accidents, and natural disasters. The Department of Homeland Security, and not the United States Department of the Interior, is equivalent to the Interior ministries of other countries.

Links

Website
Data

APIs

Department of Housing and Urban Development

The United States Department of Housing and Urban Development (HUD) is a Cabinet department in the Executive branch of the United States federal government. Although its beginnings were in the House and Home Financing Agency, it was founded as a Cabinet department in 1965, as part of the "Great Society" program of President Lyndon Johnson, to develop and execute policies on housing and metropolises.

Links

Department of Justice

The United States Department of Justice (DOJ), also known as the Justice Department, is the U.S. federal executive department responsible for the enforcement of the law and administration of justice, equivalent to the justice or interior ministries of other countries. The Department is led by the Attorney General, who is nominated by the President and confirmed by the Senate and is a member of the Cabinet. The current Attorney General is Eric Holder.

Links

APIs

Department of Labor

The United States Department of Labor (DOL) is a cabinet-level department of the U.S. federal government responsible for occupational safety, wage and hour standards, unemployment insurance benefits, re-employment services, and some economic statistics; many U.S. states also have such departments. The department is headed by the U.S. Secretary of Labor.

Links

APIs

DOL API

Department of State

The United States Department of State (DoS),[3] often referred to as the State Department, is the United States federal executive department responsible for international relations of the United States, equivalent to the foreign ministry of other countries. The Department was created in 1789 and was the first executive department established.

Links

APIs

Office of the Historian Ebook Catalog API

Department of the Interior

The United States Department of the Interior (DOI) is the United States federal executive department of the U.S. government responsible for the management and conservation of most federal land and natural resources, and the administration of programs relating to American Indians, Alaska Natives, Native Hawaiians, territorial affairs, and insular areas of the United States.

Links

APIs

Recreation Information Database (RIDB)

Department of the Treasury

The Department of the Treasury (DoT) is an executive department and the treasury of the United States federal government. It was established by an Act of Congress in 1789 to manage government revenue. The Department is administered by the Secretary of the Treasury, who is a member of the Cabinet. Jack Lew is the current Secretary of the Treasury; he was sworn in on February 28, 2013.

Links

APIs

Department of Transportation

The United States Department of Transportation (USDOT or DOT) is a federal Cabinet department of the U.S. government concerned with transportation. It was established by an act of Congress on October 15, 1966, and began operation on April 1, 1967. It is governed by the United States Secretary of Transportation.

Links

APIs

Department of Veterans Affairs (VA)

Visit Public Workspace

The United States Department of Veterans Affairs (VA) is a government-run military veteran benefit system with Cabinet-level status. With a total 2009 budget of about $87.6 billion, VA employs nearly 280,000 people at hundreds of Veterans Affairs medical facilities, clinics, and benefits offices and is responsible for administering programs of veterans’ benefits for veterans, their families, and survivors. In 2012, the proposed budget for Veterans Affairs was $132 billion.[1] The VA 2014 budget request for 2014 was $152.7 billion. This included $66.5 billion in discretionary resources and $86.1 billion in mandatory funding. The discretionary budget request represented an increase of $2.7 billion, or 4.3 percent, over the 2013 enacted level.

Links

APIs

Drug Enforcement Administration (DEA)

The Drug Enforcement Administration is a United States federal law enforcement agency under the United States Department of Justice, tasked with combating drug trafficking and distribution within the United States.

Links

Website
Data

Economic Development Administration (EDA)

The U.S. Economic Development Administration (EDA) is an agency in the United States Department of Commerce that provides grants and technical assistance to economically distressed communities in order to generate new employment, help retain existing jobs and stimulate industrial and commercial growth through a variety of investment programs.

Links

Website
Data

Economic Research Service

USDA is committed to providing safe and nutritious food, conservation, sustainable food production and revitalizing rural America.Comment Policy: http://www.usda.gov/newmedia

Links

APIs

Election Assistance Commission (EAC)

The Election Assistance Commission is an independent agency of the United States government created by the Help America Vote Act of 2002. The Commission serves as a national clearinghouse and resource of information regarding election administration.

Links

Website
Data

Employment and Training Administration (ETA)

The Employment and Training Administration is part of the U.S. Department of Labor. Its mission is to provide training, employment, labor market information, and income maintenance services.

Links

Website
Data

Environmental Protection Agency (EPA)

Visit Public Workspace

The United States Environmental Protection Agency (EPA or sometimes USEPA) is an agency of the U.S. federal government which was created for the purpose of protecting human health and the environment by writing and enforcing regulations based on laws passed by Congress. The EPA was proposed by President Richard Nixon and began operation on December 2, 1970, after Nixon signed an executive order. The order establishing the EPA was ratified by committee hearings in the House and Senate. The agency is led by its Administrator, who is appointed by the president and approved by Congress. The current administrator is Gina McCarthy. The EPA is not a Cabinet department, but the administrator is normally given cabinet rank.

Links

APIs

Farm Service Agency (FSA)

The Farm Service Agency is the United States Department of Agriculture agency that was formed by merging the farm loan portfolio and staff of the Farmers Home Administration and the Agricultural Stabilization and Conservation Service.

Links

Website

APIs

National Agriculture Imagery Program (NAIP) Status Web Service

Federal Aviation Administration (FAA)

The Federal Aviation Administration is the largest modern transportation agency and a governmental body of the United States with powers to regulate all aspects of civil aviation in that nation as well as over its surrounding international waters.

Links

Federal Bureau of Investigation (FBI)

The Federal Bureau of Investigation (FBI) is the domestic intelligence and security service of the United States, which simultaneously serves as the nation's prime federal law enforcement agency. Operating under the jurisdiction of the U.S. Department of Justice, the FBI is concurrently a member of the U.S. Intelligence Community and reports to both the Attorney General and the Director of National Intelligence. A leading U.S. counterterrorism, counterintelligence, and criminal investigative organization, the FBI has jurisdiction over violations of more than 200 categories of federal crimes.

Links

APIs

Federal Bureau of Prisons (BOP)

The Federal Bureau of Prisons is a United States federal law enforcement agency under the Department of Justice responsible for the care, custody, and control of incarcerated individuals.

Links

Website
Data

Federal Communications Commission (FCC)

Visit Public Workspace

The Federal Communications Commission (FCC) is an independent agency of the United States government, created by Congressional statute (see 47 U.S.C. § 151 and 47 U.S.C. § 154) to regulate interstate and international communications by radio, television, wire, satellite, and cable in all 50 states, the District of Columbia and U.S. territories. The FCC works towards six goals in the areas of broadband, competition, the spectrum, the media, public safety and homeland security. The Commission is also in the process of modernizing itself.

Links

APIs

Federal Deposit Insurance Corporation (FDIC)

The Federal Deposit Insurance Corporation is one of two agencies that provide deposit insurance to depositors in U.S. depository institutions, the other being the National Credit Union Administration, which regulates and insures credit unions.

Links

APIs

BankFind Suite

Federal Election Commission (FEC)

Visit Public Workspace

The Federal Election Commission is an independent regulatory agency of the United States whose purpose is to enforce campaign finance law in United States federal elections.

Links

APIs

OpenFEC

Federal Emergency Management Agency

The Federal Emergency Management Agency (FEMA) is an agency of the United States Department of Homeland Security, initially created by Presidential Reorganization Plan No. 3 of 1978 and implemented by two Executive Orders on April 1, 1979. The agency's primary purpose is to coordinate the response to a disaster that has occurred in the United States and that overwhelms the resources of local and state authorities. The governor of the state in which the disaster occurs must declare a state of emergency and formally request from the president that FEMA and the federal government respond to the disaster.

Links

Website
Data

APIs

OpenFEMA

Federal Energy Regulatory Commission (FERC)

The Federal Energy Regulatory Commission is the United States federal agency that regulates the transmission and wholesale sale of electricity and natural gas in interstate commerce and regulates the transportation of oil by pipeline in interstate commerce.

Links

Federal Highway Administration (FHWA)

The FHWA, a USDOT agency, supports state and local governments in the design, construction and maintenance of the nation's highway system, and roads on federally and tribally owned lands.

Links

Federal Housing Finance Agency (FHFA)

The Federal Housing Finance Agency (FHFA) is an independent federal agency in the United States created as the successor regulatory agency of the Federal Housing Finance Board (FHFB), the Office of Federal Housing Enterprise Oversight (OFHEO), and the U.S. Department of Housing and Urban Development government-sponsored enterprise mission team,[2] absorbing the powers and regulatory authority of both entities, with expanded legal and regulatory authority, including the ability to place government sponsored enterprises (GSEs) into receivership or conservatorship.

Links

Federal Maritime Commission (FMC)

The United States Federal Maritime Commission is an independent federal agency based in Washington, D.C. that is responsible for the regulation of oceanborne international transportation of the U.S.

Links

Federal Motor Carrier Safety Administration (FMCSA)

The Federal Motor Carrier Safety Administration is an agency in the United States Department of Transportation that regulates the trucking industry in the United States. The primary mission of the FMCSA is to reduce crashes, injuries and fatalities involving large trucks and buses.

Links

APIs

Federal Motor Carrier Safety Administration

Federal Reserve System

The Federal Reserve System is the central banking system of the United States of America. It was created on December 23, 1913, with the enactment of the Federal Reserve Act, after a series of financial panics led to the desire for central control of the monetary system in order to alleviate financial crises.

Links

APIs

Fred API

Federal Student Aid (FAFSA)

The U.S. Department of Education's office of Federal Student Aid provides more than $120 billion in financial aid to help pay for college or career school each year.

Links

Website
Data

Federal Trade Commission (FTC)

The Federal Trade Commission is an independent agency of the United States government whose principal mission is the enforcement of civil U.S. antitrust law and the promotion of consumer protection.

Links

APIs

Federal Trade Commission

Fish and Wildlife Service (FWS)

Visit Public Workspace

The United States Fish and Wildlife Service is an agency of the US federal government within the US Department of the Interior dedicated to the management of fish, wildlife, and natural habitats.

Links

APIs

FOIA.gov

The Freedom of Information Act (FOIA), 5 U.S.C. § 552, is a federal freedom of information law that allows for the full or partial disclosure of previously unreleased information and documents controlled by the United States government. The Act defines agency records subject to disclosure, outlines mandatory disclosure procedures and grants nine exemptions to the statute. It was originally signed into law by President Lyndon B. Johnson, despite his misgivings, on July 4, 1966, and went into effect the following year.

Links

Website
Data

APIs

FOIA.gov

Food and Drug Administration

The Food and Drug Administration (FDA or USFDA) is an agency of the United States Department of Health and Human Services, one of the United States federal executive departments. The FDA is responsible for protecting and promoting public health through the regulation and supervision of food safety, tobacco products, dietary supplements, prescription and over-the-counter pharmaceutical drugs (medications), vaccines, biopharmaceuticals, blood transfusions, medical devices, electromagnetic radiation emitting devices (ERED), cosmetics and veterinary products.

Links

APIs

OpenFDA

Food and Nutrition Service (FNS)

The Food and Nutrition Service is an agency of the United States Department of Agriculture. The FNS is the federal agency responsible for administering the nation’s domestic nutrition assistance programs. The service helps to address the issue of hunger in the United States.

Links

Website
Data

APIs

Kids Site Finder

Food Safety and Inspection Service (FSIS)

The Food Safety and Inspection Service, an agency of the United States Department of Agriculture, is the public health regulatory agency responsible for ensuring that United States' commercial supply of meat, poultry, and egg products is safe, wholesome, and correctly labeled and packaged.

Links

Website
Data

Foreign Agricultural Service (FAS)

The Foreign Agricultural Service is the foreign affairs agency with primary responsibility for the United States Department of Agriculture's overseas programs — market development, international trade agreements and negotiations, and the collection of statistics and market information.

Links

Website
Data

APIs

Foreign Agricultural Service Data APIs

Forest Service

To sustain the health, diversity, and productivity of the Nation’s forests and grasslands to meet the needs of present and future generations. At the heart of our agency’s mission is our purpose—the ultimate answer to why we do what we do. Everything we do—across our broad and diverse agency—is intended to help sustain forests and grasslands for present and future generations. Why? Because our stewardship work supports nature in sustaining life. This is the purpose that drives our agency’s mission and motivates our work across the agency. It’s been there from our agency’s very beginning, and it still drives us.

Links

APIs

Recreation Information Database (RIDB)

General Services Administration (GSA)

Visit Public Workspace

The General Services Administration (GSA) is an independent agency of the United States government, established in 1949 to help manage and support the basic functioning of federal agencies. The GSA supplies products and communications for U.S. government offices, provides transportation and office space to federal employees, and develops government-wide cost-minimizing policies, and other management tasks.

Links

APIs

HealthData.gov

Welcome to HealthData.gov! This site is dedicated to making high value health data more accessible to entrepreneurs, researchers, and policy makers in the hopes of better health outcomes for all. In a recent article, Todd Park, United States Chief Technology Officer, captured the essence of what the Health Data Initiative is all about and why our efforts here are so important.

Links

Website

APIs

HealthData.gov

HistoryAtState

The Office of the Historian is an office of the United States Department of State within the Bureau of Public Affairs. The Office is responsible, under law, for the preparation and publication of the official historical documentary record of U.S. foreign policy in the Foreign Relations of the United States series. It researches and writes historical studies on aspects of U.S. diplomacy for use by policymakers in the Department and in other agencies as well for the public.

Links

Institute of Museum and Library Services (IMLS)

The Institute of Museum and Library Services is an independent agency of the United States federal government established in 1996.

Links

International Trade Administration (ITA)

Visit Public Workspace

The International Trade Administration (ITA) strengthens the competitiveness of U.S. industry, promotes trade and investment, and ensures fair trade through the rigorous enforcement of our trade laws and agreements. ITA works to improve the global busi...

Links

APIs

Library of Congress (LOC)

The Library of Congress is the largest library in the world, with millions of books, recordings, photographs, newspapers, maps and manuscripts in its collections. The Library is the main research arm of the U.S. Congress and the home of the U.S. Copyright Office.

Links

APIs

Maritime Administration

The United States Maritime Administration is an agency of the United States Department of Transportation. Its programs promote the use of waterborne transportation and its seamless integration with other segments of the transportation system, and the viability of the U.S. merchant marine.

Links

Website
Data

Millennium Challenge Corporation (MCC)

The Millennium Challenge Corporation is a bilateral United States foreign aid agency established by the U.S. Congress in 2004, applying a new philosophy toward foreign aid. It is an independent agency separate from the State Department and USAID.

Links

National Aeronautics and Space Administration

The National Aeronautics and Space Administration (NASA) is the United States government agency that is responsible for the civilian space program as well as for aeronautics and aerospace research. President Dwight D. Eisenhower established the National Aeronautics and Space Administration (NASA) in 1958[5] with a distinctly civilian (rather than military) orientation encouraging peaceful applications in space science. The National Aeronautics and Space Act was passed on July 29, 1958, disestablishing NASA's predecessor, the National Advisory Committee for Aeronautics (NACA).

Links

APIs

NASA Open APIs

National Agricultural Statistics Service (NASS)

The National Agricultural Statistics Service is the statistical branch of the U.S. Department of Agriculture and a principal agency of the U.S. Federal Statistical System.

Links

APIs

National Archives and Records Administration

Visit Public Workspace

The National Archives and Records Administration (NARA) is an independent agency of the United States government charged with preserving and documenting government and historical records and with increasing public access to those documents, which comprise the National Archives.[6] NARA is officially responsible for maintaining and publishing the legally authentic and authoritative copies of acts of Congress, presidential proclamations and executive orders, and federal regulations. The NARA also transmits votes of the Electoral College to Congress.

Links

Website

APIs

National Center for Education Statistics (NCES)

The National Center for Education Statistics is the part of the United States Department of Education's Institute of Education Sciences that collects, analyzes, and publishes statistics on education and public school district finance information in the United States.

Links

Website
Data

APIs

EDGE Open Data APIs

National Geospatial-Intelligence Agency (NGA)

Visit Public Workspace

The National Geospatial-Intelligence Agency is a combat support agency under the United States Department of Defense and a member of the United States Intelligence Community, with the primary mission of collecting, analyzing, and distributing geospatial intelligence in support of national security.

Links

APIs

National Institute of Standards and Technology (NIST)

Visit Public Workspace

The National Institute of Standards and Technology is a physical sciences laboratory and a non-regulatory agency of the United States Department of Commerce. Its mission is to promote innovation and industrial competitiveness.

Links

APIs

NIST NERDm Schema

National Institutes of Health

Visit Public Workspace

The National Institutes of Health (NIH) is a biomedical research facility primarily located in Bethesda, Maryland, USA. An agency of the United States Department of Health and Human Services, it is the primary agency of the United States government responsible for biomedical and health-related research. The NIH both conducts its own scientific research through its Intramural Research Program (IRP) and provides major biomedical research funding to non-NIH research facilities through its Extramural Research Program. With 1,200 principal investigators and more than 4,000 postdoctoral fellows in basic, translational, and clinical research, the IRP is the largest biomedical research institution on Earth, while, as of 2003, the extramural arm provided 28% of biomedical research funding spent annually in the US, or about US$26.4 billion.

Links

Website
Data

APIs

National Oceanic and Atmospheric Administration (NOAA)

Visit Public Workspace

The National Oceanic and Atmospheric Administration is an American scientific agency within the United States Department of Commerce that focuses on the conditions of the oceans, major waterways, and the atmosphere.

Links

APIs

National Park Service (NPS)

This API is designed to provide authoritative National Park Service (NPS) data and content about parks and their facilities, events, news, alerts, and more. Explore the NPS API below and even try to make API calls. In order to try an API call, you'll need to click on the "Authorize" button below and add your API key.

Links

APIs

National Park Service

National Renewable Energy Laboratory

The National Renewable Energy Laboratory (NREL) is the nation's primary laboratory for renewable energy and energy efficiency research and development (R&D).

Links

APIs

National Renewable Energy Lab

National Science Foundation

The National Science Foundation (NSF) is a United States government agency that supports fundamental research and education in all the non-medical fields of science and engineering. Its medical counterpart is the National Institutes of Health. With an annual budget of about US$7.0 billion (fiscal year 2012), the NSF funds approximately 20% of all federally supported basic research conducted by the United States' colleges and universities. In some fields, such as mathematics, computer science, economics and the social sciences, the NSF is the major source of federal backing.

Links

Website
Data

APIs

NSF Awards API

National Security Agency (NSA)

The National Security Agency (NSA) is an intelligence organization of the United States government, responsible for global monitoring, collection, and processing of information and data for foreign intelligence and counterintelligence purposes – a discipline known as signals intelligence (SIGINT). NSA is concurrently charged with protection of U.S. government communications and information systems against penetration and network warfare.

Links

National Telecommunications and Information Administration (NTIA)

The National Telecommunications and Information Administration (NTIA) is an agency of the United States Department of Commerce that serves as the President's principal adviser on telecommunications policies pertaining to the United States' economic and technological advancement and to regulation of the telecommunications industry.

Links

Website
Data

National Transportation Safety Board (NTSB)

The National Transportation Safety Board is an independent U.S. government investigative agency responsible for civil transportation accident investigation.

Links

Natural Resources Conservation Service (NRCS)

Natural Resources Conservation Service, formerly known as the Soil Conservation Service, is an agency of the United States Department of Agriculture that provides technical assistance to farmers and other private landowners and managers.

Links

APIs

National Water and Climate Center AWDB database

Nuclear Regulatory Commission

The Nuclear Regulatory Commission (NRC) is an independent agency of the United States government, established by the Energy Reorganization Act of 1974, and began operations on January 19, 1975. As one of two successor agencies to the United States Atomic Energy Commission, the NRC's role is to protect public health and safety related to nuclear energy. It oversees reactor safety and security, reactor licensing and renewal, licensing of radioactive materials, radionuclide safety, and spent fuel management including storage, security, recycling, and disposal.

Links

APIs

Adams

Occupational Safety and Health Administration (OSHA)

The Occupational Safety and Health Administration is a large regulatory agency of the United States Department of Labor that originally had federal visitorial powers to inspect and examine workplaces.

Links

Website
Data

Office of Justice Programs

The Office of Justice Programs (OJP) is an agency of the United States Department of Justice that focuses on crime prevention through research and development, assistance to state and local law enforcement and criminal justice agencies through grants, and assistance to crime victims.

Links

Office of Personnel Management

The United States Office of Personnel Management (OPM) is an independent agency of the United States government that manages the civil service of the federal government.

Links

Website
Data

APIs

Federal Government Operating Status

Search.gov

Powering over 2,000 search boxes on Federal websites. It would be impossible to match the value of GSA’s Search service by procuring, building, and configuring a custom solution ourselves. With the combination of a very feature-rich search service and knowledgeable staff, this is the most appealing search tool for government websites.

Links

Website

APIs

Securities and Exchange Commission (SEC)

The U.S. Securities and Exchange Commission is a large independent agency of the United States federal government that was created following the stock market crash in the 1920s to protect investors and the national banking system.

Links

Small Business Administration

The Small Business Administration (SBA) is a United States government agency that provides support to entrepreneurs and small businesses. The mission of the Small Business Administration is "to maintain and strengthen the nation's economy by enabling the establishment and viability of small businesses and by assisting in the economic recovery of communities after disasters". The agency's activities are summarized as the "3 Cs" of capital, contracts and counseling.

Links

Website
Data

APIs

SBA PPP Loan Forgiveness

Small Business Innovation Research

The Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs are highly competitive programs that encourage domestic small businesses to engage in Federal Research/Research and Development (R/R&D) with the potential for commercialization. Through a competitive awards-based program, SBIR and STTR enable small businesses to explore their technological potential and provide the incentive to profit from its commercialization. By including qualified small businesses in the nation's R&D arena, high-tech innovation is stimulated, and the United States gains entrepreneurial spirit as it meets its specific research and development needs.

Links

Website

APIs

Substance Abuse and Mental Health Services Administration (SAMHSA)

The Substance Abuse and Mental Health Services Administration (SAMHSA; pronounced /ˈsæmsə/) is a branch of the U.S. Department of Health and Human Services. It is charged with improving the quality and availability of treatment and rehabilitative services in order to reduce illness, death, disability, and the cost to society resulting from substance abuse and mental illnesses. The Administrator of SAMHSA reports directly to the Secretary of the U.S. Department of Health and Human Services. SAMHSA's headquarters building is located outside of Rockville, Maryland.

Links

Tennessee Valley Authority (TVA)

The Tennessee Valley Authority (TVA) is a federally owned corporation in the United States created by congressional charter on May 18, 1933, to provide navigation, flood control, electricity generation, fertilizer manufacturing, and economic development to the Tennessee Valley, a region particularly affected by the Great Depression. Senator George W. Norris (R-Nebraska) was a strong sponsor of this project. TVA was envisioned not only as a provider, but also as a regional economic development agency that would use federal experts and rural electrification to help modernize the rural region's economy and society.

Links

U.S. Department of State Office of the Historian

The Office of the Historian is staffed by professional historians who are experts in the history of U.S. foreign policy and the Department of State and possess unparalleled research experience in classified and unclassified government records. The Office’s historians work closely with other federal government history offices, the academic historical community, and specialists across the globe. The Office is directed by The Historian of the U.S. Department of State.

Links

U.S. Geological Survey (USGS)

Visit Public Workspace

The United States Geological Survey (USGS, formerly simply Geological Survey) is a scientific agency of the United States government. The scientists of the USGS study the landscape of the United States, its natural resources, and the natural hazards that threaten it. The organization has four major science disciplines, concerning biology, geography, geology, and hydrology. The USGS is a fact-finding research organization with no regulatory responsibility.

Links

APIs

United States Citizenship and Immigration Services (USCIS)

U.S. Citizenship and Immigration Services is an agency of the United States Department of Homeland Security that administers the country's naturalization and immigration system.

Links

United States Immigration and Customs Enforcement (ICE)

The U.S. Immigration and Customs Enforcement is a federal law enforcement agency under the U.S. Department of Homeland Security. ICE's stated mission is to protect the United States from the cross-border crime and illegal immigration that threaten national security and public safety.

Links

Website
Data

United States International Trade Commission (USITC)

The United States International Trade Commission is an independent, bipartisan, quasi-judicial, federal agency of the United States that provides trade expertise to both the legislative and executive branches.

Links

Website
Data

United States Patent and Trademark Office (USPTO)

Visit Public Workspace

The United States Patent and Trademark Office (PTO or USPTO) is an agency in the U.S. Department of Commerce that issues patents to inventors and businesses for their inventions, and trademark registration for product and intellectual property identification. The USPTO is "unique among federal agencies because it operates solely on fees collected by its users, and not on taxpayer dollars". Its "operating structure is like a business in that it receives requests for services—applications for patents and trademark registrations—and charges fees projected to cover the cost of performing the services it provides.

Links

APIs

USPTO APIs

United States Postal Service (USPS)

The United States Postal Service is an independent agency of the executive branch of the United States federal government responsible for providing postal service in the United States, including its insular areas and associated states.

Links

APIs

USPS API

USDA Rural Development (RD)

The United States Department of Agriculture, also known as the Agriculture Department, is the U.S. federal executive department responsible for developing and executing federal laws related to farming, forestry, rural economic development, and food.

Links

Website
Data

This represents about two days worth of work, which is also upon some work I've done over the years. There are definitely more APIs out there that I couldn't easily catalog, but I've begun setting up workspaces to help me continue building on this work in an ongoing, community driven way. This sprint is just meant to get all of these federal agencies loaded up in my brain again, stir up thoughts about what is happening, and push me to propose some next steps for specific agencies, but also the overall API conversation. After doing this work here are a few of my thoughts about what is going on and what might come next.

Growth In APIs - There are definitely more APIs that when I last looked across agencies, and there are some agencies you can tell have embraced an API way of doing things.
Lots of 404s - There are a lot of 404s when I look through legacy lists of APIs I was familiar with, and it can be tough to actually tell if there is anyone home behind some of the APIs.
Templates Matter - The agencies who have provided portal templates, and open blueprints for publishing APIs makes a big difference, making their APIs much more consistent and usable.
Swagger Usage Has Grown - There is a significant uptick in the usage of Swagger across agencies, including adoption of OpenAPI 3.0, demonstrating the value of the specifications across government APIs.
Huge Data Opportunity (As Always) - I included a link to any agency's data landing page because some of them have APIs I may not have indexed yet, but also because they represent the API opportunity across data.
Standardized API Management - The usage of API Umbrella by GSA and it's standardized usage across agencies has made an impact on working with APIs, and reduced friction when it comes to onboarding with APIs.

I went into this work fairly depressed about what I might find at the close of the current administration, but I left fairly optimistic about the work that is going on across agencies. Don't get me wrong, the overall API conversation across federal government is still in a pretty sad state. There is a massive wealth of data resources available, but discovery and integration with these datasets are not at all standardized, and there are few of the affordances we often take for granted in the private sector. I always view federal government as being a minimum of 5-10 years beyond what is going on in the private sector, but we are going to need a huge amount of investment in APIs across federal agencies if they are going to play catch up, let alone begin to lead in the API conversation.

Getting to Work on APIs Across Federal Agencies

I could spend months profiling the APIs that exist across the agencies profiled above. I could spend 1000x that amount of time turning the data resources available across these federal agencies into APIs. I have begun setting up workspaces for the agencies who had Swagger and OpenAPI available, while also allowing me to begin reverse engineering some of the easier APIs using Postman Interceptor -- helping me create collections and OpenAPIs just by using an API. I will be spending time each week working within the public workspace fore each of the federal agencies and begin setting up new workspaces for some of the other agencies. I finally have the time, resources, and the platform to begin making a dent in this work. Historically I have been profiling APIs for federal agencies, but it was more just about bookmarking whatever I have found and organizing them into lists. Now, with Postman public workspaces I can conduct this work out in the open while also potentially sharing the load with the community, while getting more precise in how I define and map out each of the APIs using the Postman platform.

GitHub has made a massive impact on the federal government over the last decade and I am confident that Postman public workspaces are going to take the API conversation across federal agencies to similar levels. I can keep mapping out the APIs that exist, which is something I will keep doing, but I also want to provide resources to help the people doing the hard-work behind these APIs and datasets understand the benefits of publishing consistent, simple, and usable APIs by default. There is too much work out there for use to do after the fact, or from the outside-in. Ultimately we need the stewards of all of this valuable information to be API literate, understanding what is possible when we are publishing APIs in a consistent way, and help them get beyond talking about it, and actually doing it. It is extremely important that federal agencies not just do APIs across the board, but also be there to lead conversations. Similar to what we are seeing with the Fast Healthcare Interoperability Resources (FHIR). While there is still a huge amount of work to be done here, the API specification is finally seeing the type of industry wide adoption needed to make an impact on the healthcare space. We need to begin doing this for every other industry, helping stabilize and standardize the API conversation across all of the top business sectors. The most important investment our federal government can make in the next administration is to invest in API education and training across the board, taking the lead when it comes to the valuable resources our economy need to stay ahead of the curve.

A Postman Collection For Capitalizing Folders Requests In Collections

Thu, 17 Dec 2020 00:00:00 +0000

published: true

title: ‘A Postman Collection For Capitalizing Folders & Requests In Collections’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/postman_collection_for_capitalizing_folders_requests_in_collections.png

Sometimes I need to do bulk updates to Postman collections and there is no better way to automate this than to use a Postman Collection that uses the Postman API—inception level stuff, so be careful ;-). I have setup a dedicated public workspace where I am building out these utility type operational level collections that help me manage the API lifecycle out ahead of what the Postman GUI is capable of doing. Some of the things I am doing will eventually make its way to the Postman UI, but some of them will not. Either way, I am too impatient to wait, and one of the things I love about Postman is that I can hack my way forward through just about any situation using the Postman API.

Building on my base collection for pulling and updating collections, I added two differents requests that will help me capitalize the first letter of each word in a folder or request name of a collection.

To run, all you have to do is make sure you’ve entered a collection ID (pulled from URL or info tab), and hit run—it will loop through each folder and request and capitalize the words, and then save the collection using the Postman API. So you can immediately put to use the same collection with the changes you desired.

If you have any questions on the collection for pulling a collection and making changes using the Postman API, feel free to submit a comment for the collection as part of the collection workspace. I’ll be centralizing the evolution of this collection, as well as other collection related collections within this workspace. You can also fork the collection and use in your workspaces, and submit back any enhancements you’d like to see as a pull request. If you have any questions that you don’t want to see in the comments for the collection, feel free to email me at info@apievangelist.com.

Apis Are At The Center Of The Federal Trade Commission Ftc Lawsuit Against Facebook

Sat, 12 Dec 2020 00:00:00 +0000

published: true

title: ‘APIs are at the Center of the Federal Trade Commission (FTC) Lawsuit Against Facebook’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/federal_trade_commission_facebook_filing.png

The Federal Trade Commission is sueing Facebook, alleging that they are illegally maintaining a monopoly on the personal social network space using a continued series of anticompetitive behavior. The suit includes a coalition of attorneys general of 46 states, bringing the latest wave of regulatory scrutiny into the social media platform, highlighting three main dimensions of how Facebook has engaged in a systematic strategy of anticompetitive practices that has resulted in their current dominant position online.

Instagram - Facebook purchased Instagram to remove what they saw as one of the biggest threats they faced, while also purchasing market dominance in the image and photos sharing space.
WhatsApp - Fsacebook purchased WhatsApp to remove what they saw as one of the biggest threats they faced, while also purchasing market dominance in the messaging space.
API - Facebook used their API to cut off access to applications they saw as a threat, and to generally suffocate anything else they saw as a threat to their business model.

The anticompetitive nature of Facebook’s Instagram and WhatsApp purchase isn't really in my wheelhouse, but how the API was used for anticompetitive activity most definitely is my speciality. I am happy to see the FTC finally elevate the abuse that has been going on at Facebook via it’s APIs, not just because of the impact on the Facebook developer community, but more importantly the wider tech sector. Facebook’s approach hurts wider competition across the tech sector, but also hurts the space API sector--ultimately giving APIs a bad reputation. To help me speak intelligently to what is occurring as part of the FTC’s lawsuit, offer my advice on what a remedy might be, while also contributing to the future of regulation in the tech sector, I wanted to gather my thoughts in a post here on the blog.

Anticompetitive Conditioning Using The Facebook API

The acquisition of Instagram and WhatsApp by Facebook to neutralize threats in two keys areas are a critical part of this filing and Facebook’s strategy to dominate, but it is their API that has the biggest impact in suffocating out competition in the areas of photos and messaging, but every other dimension in which other companies can compete with Facebook via their own API. I think the FTC filing sums it up best in section 22 with the phrase “anticompetitive conditioning”

22. Anticompetitive Conditioning. In addition to its strategy of acquiring competitive threats to its personal social networking monopoly, Facebook has, over many years, announced and enforced anticompetitive conditions on access to its valuable platform interconnections, such as the application programming interfaces (“APIs”) that it makes available to third-party software applications.

This sums up the slow roll suffocation of competition that exists as part of the politics of APIs across not just Facebook, but also other leading platforms via their API ecosystems. Companies who view their API developer ecosystems as an asset use the API management layer of their APIs to incentivize developers, where companies like Facebook who have reached a size and scope where they view a significant portion of their API consumers as a threat will choose to engage in widespread anticompetitive conditioning via the management layer of their APIs.

Removing Facebook API Access for the Competition

The FTC filing focuses on the most overt and visible anticompetitive behavior from Facebook at the API layer, in particular the turning off of access to API consumers when they are viewed as a threat to their platform business. It is commonplace that API consumers have to register and obtain keys for access to API resources. This is a known aspect of API management, the API provider and consumer dance on Facebook and beyond, and it is common to revoke the API access keys of consumers who violate your terms of service. However, many API providers like Facebook also use this practice to “condition the competition”. The FTC filing focuses on two main consumers who Facebook cut off access, first in section 153 with Path.

153. First, Facebook targeted promising apps that provided personal social networking. For example, Facebook took actions against a personal social networking competitor, Path, which was founded by a former Facebook manager. In or around April 2013, Facebook terminated Path’s access to Facebook’s APIs, and Path’s growth subsequently slowed significantly.

After Path, there was another high profile competitor (Twitter) that was moving into their territory, which the FTC filing outlined in section 155.

155. Similarly, in January 2013, Facebook’s Director of Platform Partnerships and Operations wrote to colleagues: “[T]witter launched Vine today which lets you shoot multiple short video segments to make one single, 6-second video. As part of their NUX [new user experience], you can find friends via FB. Unless anyone raises objections, we will shut down their friends API access today. [W]e’ve prepared reactive PR, and I will let Jana know our decision.” Mr. Zuckerberg replied: “[Y]up, go for it.” By cutting off Vine, Facebook prevented it from accessing APIs that would have helped it grow.

Path and Vine were just the tip of the iceberg when it comes to Facebook’s campaign to cut off access to competitors when it came to their APIs, as the FTC filing outlines in section 156.

156. The third group of targets were promising apps that offered mobile messaging services, that were existing competitors of Facebook Messenger, and that ultimately threatened to develop into competitive threats to Facebook Blue. Throughout 2013, Facebook blocked mobile messaging apps from using commercially significant APIs. For example, in August 2013, Facebook undertook an enforcement strike against a number of messaging apps simultaneously, with the Head of Developer Enforcement directing colleagues to restrict them from “accessing any read APls beyond basic info[,]” instructing that “we will not be communicating with the [developers] in any way about these restrictions.”

This reflects direct action taken by Facebook to snuff out the competition by removing access to their increasingly valuable resources by the competition. It becomes much more difficult to measure the indirect chilling effect all of this had on developers within the Facebook API ecosystem who would self-censor, and avoid developing applications that may get the attention of Facebook, even if they weren’t directly competing with the giant. Sometimes you just didn’t know if what you were building was emulating Facebook features and it would be better to just not build your application at all.

Anticompetitive Tones Set By API Terms of Service

The terms of service (TOS) for API platforms always set the tone for engagement between API provider and consumer, but it is something that Facebook has wielded with great success when it comes to their anticompetitive practices across their developer community. There are several consecutive sections of the FTC filing that highlight the chilling effects TOS changes have had to the Facebook developer ecosystem,

138. From July 2011 until December 2018, Facebook publicly announced and imposed a set of anticompetitive conditions governing access to Facebook Platform.

139. On July 27, 2011, Facebook introduced a new policy that “Apps on Facebook may not integrate, link to, promote, distribute, or redirect to any app on any other competing social platform.”

141. Following that, Facebook imposed several other policies restricting app developers’ use of Facebook Platform, including Facebook APIs. Through these policies, Facebook used its control over APIs to deter and suppress competition.

142. September 2012: no exporting data to competitor social networks. On September 12, 2012, Facebook introduced a new policy: “Competing social networks: You [developers] may not use Facebook Platform to export user data into a competing social network without our permission[.]”

143. January 2013: no promotion / data export to any app that “replicates a core Facebook product or service.” On January 25, 2013, Facebook added a new condition to prevent developers from “replicating core functionality” (i.e., competing with Facebook), or assisting others who might do so: Reciprocity and Replicating core functionality: (a) Reciprocity: Facebook Platform enables developers to build personalized, social experiences via the Graph API and related APIs. If you use any Facebook APIs to build personalized or social experiences, you must also enable people to easily share their experiences back with people on Facebook. (b) Replicating core functionality: You may not use Facebook Platform to promote, or to export user data to, a product or service that replicates a core Facebook product or service without our permission. (Emphasis added.)

These types of TOS practices make for a very confusing environment within the Facebook developer ecosystem. As a developer using the API to develop an application you are faced with the task of trying to speculate on what constitutes a competing feature and what does not in a very fast moving sector. It also gives Facebook a pretty wide spectrum to choose from when it comes to determining which API consumers have violated their TOS, and which have not. Making for a pretty fraught relationship when it comes to building on the Facebook platform, and a pretty terrifying uphill battle when it comes to actually taking on the tech giant when it comes to any aspect of their current or future business.

The Role APIs Have Played in Facebook’s Dominance

The FTC filing does a great job outlining the role that the API community has played in Facebook’s evolution as a platform, and the role the platform plays in developing web and mobile applications across almost every business sector. As outlined in section 135 the API community was the vehicle for Facebook growth over the years and resulting dominance.

134. Further, third-party apps helped Facebook grow and provided other forms of value to Facebook by promoting Facebook around the internet, via the Facebook plug-ins and by directing social data, such as “Likes,” back to Facebook Blue. Since launching its Facebook Platform and Open Graph initiatives, Facebook has grown significantly, adding at least 150 million monthly users each year and serving over 2.7 billion monthly users worldwide today.

In the early years of Facebook, as with any platform, they needed their API developer community to build out new features, and deliver critical integrations with other platforms. But as the power of Facebook grew, it slowly, and then more rapidly began to assert the power it has over the trajectory of applications being developed by the community, as outlined in section 135.

135. With the success of Facebook Platform, Facebook became important infrastructure for third-party apps and obtained immense power over apps’ developmental trajectories, competitive decision-making, and investment strategies.

Fully aware of it’s increased power, and growing even more hungry for power, Facebook would begin to see innovation within its own community as a threat, rather than an asset as it had in earlier days. Weaponizing the very thing that had helped it grow from the tech giant it is today, eliminating anything it saw as a threat within its own community.

151. Moreover, Facebook knew and expected that API access was sufficiently important to affect the incentives of developers and the developmental trajectories of their apps. Developers were incentivized to make decisions that would not jeopardize their access to Facebook’s APIs. An internal Facebook slide deck dated January 2014 dealing with Facebook Platform policies directly acknowledged the importance of API access, asking whether Facebook was “[c]omfortable altering / killing prospects of many startups[.]”

In the early days of any platform your API management layer is designed to secure your digital resources, requiring developers to register and obtain access keys before they can begin making API calls from their applications. API management is about striking a balance between API provider and consumer, agreeing to fair and equitable terms of service, as well as sometimes a price for providing access to digital resources and capabilities. It is how much of the growth we’ve seen in Facebook, Twitter, and other leading platforms has been built out, but once a platform has seen enough growth, investment, and power accrue, there always seems to come a time where leaders begin to see the community as a threat--which is something Facebook has taken to entirely new levels.

The Weaponization of API Change Management

One of the greatest challenges any online platform or digital service will face comes in the form of managing change. Platform code and APIs are perpetually evolving, requiring some fairly advanced strategies for dealing with change using a well planned road map, while also actively addressing technical debt that accrues as part of legacy code that must be either evolved or deprecated. Version control is one way that platforms help manage and communicate change when it comes to making resources available via web APIs, helping manage the forward motion of a platform, while keeping API consumers moving along at the same speed. Change management is something Facebook has employed, but due to the overall tone being set by their leadership, they have also managed to weaponize something that is usually not so politically charged, as outlined in section 146.

146. May 2014: Platform 3.0 launches. A new approach—dubbed Platform 3.0—launched on May 2, 2014. At that time, Facebook terminated all apps’ access to certain APIs, which included restricting third-party apps from accessing information about their users’ friends who were not already using that third-party app.

Major releases are not that unusual, but deprecating legacy resources and forcing developers to migrate to a major release is less common. Versioning and change management is something that is hard under any circumstance, requiring a great deal of communication between API provider and consumer. It is something that is often further balanced by giving developers an extended period of time to make a migration, before older APIs are deprecated. Facebook is notorious in the API industry for committing many of the sins that we collectively agree upon in the industry should be avoided when it comes to striking a balance in our API ecosystems.

Communication - Facebook long has a reputation of poor communication around it’s releases, ensuring API consumers are prepared for the future, and are given enough time to evolve their integrations to keep pace with the removal of services. Communication is essential for helping keep an API developer community moving at the same pace as the platform, and is something that can easily be weaponized when it comes to keeping some partners informed of what is happening, while keeping others in the dark, making it difficult for competitors to keep up.
Velocity - Mark Zuckerberg crafted the phrase, “Move fast and break things. Unless you are breaking stuff, you are not moving fast enough.” This mantra is reflected in how they run their APIs, moving fast, making lots of changes, and quickly disposing of technical debt. This is something that makes it difficult for the API developer community to keep pace with, especially without proper communication and notification of what the future may hold.
Breaking Changes - One of the core tenets of API development is that you do not introduce breaking changes in the design of your API with a release, unless it is deemed a major release (ie. 1.0, 2.0, 30). Again, Facebook is notorious for releasing mysterious breaking changes as part of their release process without much communication between why, breaking integrations along the way. When you are the one breaking things it is easy to make sure your applications and integrations keep up, but if you aren’t, your applications are always going to be left behind and be unreliable.
Deprecation - API providers have different views of how long you keep older versions of APIs around before you deprecate them. Google often gives 18 months, while others will support in perpetuity -- I believe Salesforce still supports many of its original APIs. However, Facebook regularly deprecates older versions of APIs shortly after the release of a newer version, giving developers months, weeks, or no notice before an API disappears, as Facebook did with their Audience Insight API in 2017, giving no heads up that the controversial API would disappear, something you won’t find in their road map or change log -- it just went away after regulatory scrutiny.

Being able to deal with change is how you remain competitive. Weaponizing change on your platform so that you can punish and introduce friction for competitors is how you unfairly stay ahead in a fast moving game. Facebook’s anticompetitive practices were not limited to cutting off access to APIs, and setting an anticompetitive tone within the API community. It is also about weaponizing change and making it impossible for not just the competition to keep up, but also making it difficult for other application and integration developers to even get a foothold with their new and novel contributions to the market--something that hurts everyone, including Facebook.

Finding a Remedy for Anticompetitive Behavior at the API Layer

Facebook’s anticompetive behavior is not original. It is a common abuse amongst platforms, and something I’ve seen and written about numerous times over the last decade. Facebook’s sins can be also seen on Twitter,and other platforms whose business models rely on advertising, rather than subscription or more direct approaches to revenue. The stakes are always higher when a platform is built using user-generated content and monetized via eyeballs and engagement. These are always the platforms whose APIs are wide open in the early days, and become more locked down and exploitative once the value, importance, and investment increases. APIs are a great way to build a platform, while incrementally over time ensuring that all or most of the value flows in your direction, and exploiting developers and end-users for the labor needed to generate all of the value.

As stated by section 29 in the FTC filing, Facebook has been savvy in backing off or at least being more secretive about some of its anticompetitive practices, and even if there are repercussions for their acquisition of Instagram and WhatsApp, there is nothing stopping Facebook from continuing with their strategy, and finds ways to morph their anticompetive behavior by obfuscating it at the API layer.

29. Today, Facebook’s course of conduct to unlawfully maintain its personal social networking monopoly continues, and must be enjoined. Facebook continues to hold and operate Instagram and WhatsApp, and continues to keep them positioned to provide a protective “moat” around its personal social networking monopoly. Facebook continues to look for competitive threats, and will seek to acquire them unless enjoined. Likewise, Facebook’s imposition of anticompetitive conditions on APIs continued until suspended—at least for the time being—in the glare of attention from governments and regulators around the globe. Facebook will resume those policies or equivalent measures unless enjoined.

Facebook is in the business of constant evolution. It will keep morphing and evolving its strategy for dominating the personal social network, photo sharing, messaging, and any other dimensions it sees a key to its success. Facebook’s API layer has been how the platform has done this in the past, and it will be key to how it does it in the future. APIs are how Salesforce, Amazon, Google, Twitter, and others have built their platforms, and it is how they, and other tech platforms deal with change, remain agile and nimble, and remain competitive in a digital landscape. To help make sense of this fast moving, ever changing digital landscape you have to use the same tools that these platforms are employing to make sense of what is happening. You have to employ the very same tooling Facebook and these providers are using to their advantage, tapping into the APIs, and more specifically the API management layer that already exists to begin stabilizing the Internet using regulation in the same ways we stabilized energy, telecommunication, automobile, and other critical industries our economy and society depends upon.

A Quick Primer on API Management

The infrastructure Facebook uses to manage developer access to their APIs is widely known as API management infrastructure across the tech sector. API management began as a formal approach to managing API consumption in 2006 with the introduction of a startup called Mashery, which was eventually sold to Intel, and then to Tibco. Shortly after Mashery launched a wave of API management providers emerged to help companies secure and manage APIs at scale. After some consolidation of API management providers by 2015, API management has quickly become something that is baked into the fabric of the cloud, and is a fundamental offering on Amazon Web Services, Microsoft’s Azure, Google, IBM, and other cloud infrastructure platforms.

API management is how you secure your digital assets and capabilities as an enterprise organization. It is how you strike a balance between making these resources accessible on the Internet and being able to retain control over who has access to these resources and the value that is generated around them. API management is defined by a couple of key features that you will find played a key role in shaping Facebook’s anticompetitive strategy.

Accounts - Every developer is required to signup for an account if they expect to have access to an API, providing a minimum of name and email before they can gain access to any valuable API resources.
Applications - Every developer is required to submit a definition of their application, providing a name and description of its purpose at a minimum, outlining what each developer will be building on top of an API.
Keys - Once an account and application is approved, a set of keys are issued to a developer. These keys must accompany every request made to an API, providing essentially a user and password that authenticates a user, but also can be used to aggregate and report of usage by each developer.
Plans - APIs can be organized into different access tiers or plans by the API provider. Providing a way to standardize access to digital resources and capabilities by different groups of API consumers. Defining what can be accessed by consumers, and what the business model (or lack of one) that will exist as part of the agreement between API provider and consumer.
Rate Limiting - All API access is rate limited, allowing consumers to only access what their plan or access tier will allow. Ensuring that API consumers don’t overuse and abuse access to digital resources, without it being in alignment with the platform's goals, while also protecting end-users.
Logging - All API calls are logged, including the API consumers keys used to make each call, and the fingerprint of the application behind the consumption. Keeping a record of every call made to each API, and the consumer application behind each request.
Reporting - API management is all about providing awareness of who is accessing what API resources, allowing both the API provider and in some cases the API consumer a dashboard of how APIs are being put to use, which resources are being access, and a whole range of other data points that are gathered on a regular basis as part of API operations.

API management isn’t some new technological trend. It is a fundamental part of the cloud and how digital resources are made available in web, mobile, and device applications today. It is how digital resources are made available, while also keeping API consumers honest about how they are putting digital resources to work in their applications. API management quantifies and helps us visualize the value that is generated via platforms, measuring API activity in real time. API management is used to strike a balance with your API developer ecosystem, keeping a platform growing and evolving. It can also, as Facebook has demonstrated, be used to identify threats within your own ecosystem, then shut down or at least introduce friction into the applications you feel are a threat to your operations, making API management where you pull the knobs and levers of your anticompetitive digital strategy.

API Management Driving Anticompetitive Behavior

I have been studying APIs and API management for a decade now. I have seen this layer of platforms be used in many creative and innovative ways. I have also seen this layer of platforms be used in many anticompetitive ways. The API management layer is how Facebook sees the threats that exist on it’s own platform, and ultimately how it punishes it’s competitors. The API management layer is how Facebook requires API consumers to agree with it’s terms of service, and how it enforces it’s TOS. It is where Facebook decides to deprecate older versions of it’s APIs, while forcing everyone to migrate to newer versions. The API management layer is how, as the FTC filing states, Facebook is applying the anticompetitive conditioning portion of their business strategy. While the acquisition of Instagram and WhatsApp are the most visible outcomes of Facebook’s anti competitive strategy, the API management layer is where most of this behavior is playing out, and it where it will continue to morph and shape shift its strategy in coming years.

Lack of Observability Around API Access Onboarding

Facebook knows exactly how many developers it has access to their APIs. They also know exactly how many applications are accessing their APIs. The wider Facebook community, and government entities have no visibility into who does or does not have access to Facebook resources and capabilities via their APIs. There is no official accounting of who uses Facebook APIs outside of Facebook, helping shine a light on which countries applications operate in, what industries they are applied to, or any other insights about what the landscape, let alone the competitive landscape looks like across the Facebook ecosystem. All of this information exists at the API management layer within Facebook, it is just that there is no motivation for Facebook to share this information with the public, or with regulatory authorities.

Lack of Observability Around API Access Termination

Like onboarding with APIs, there is no observability into who gets their access to the Facebook APIs terminated. There is no accounting of who was terminated or why. All of this information exists at the API management layer at Facebook, but there is no incentive for Facebook to share this publicly or with regulators. There is no official accounting of who loses access to Facebook API, or whether or not this termination was justified or fair. There is no due process or official approach to contesting your loss of access to Facebook APIs. It is completely up to Facebook to terminate, and up to them whether they want to provide a reason, or whether the reason was honest and fully communicated. The API management layer provides us with everything we need to quantify whether Facebook, or any other platform is being fair and balanced in the termination of access to their APIs, we just need the will to make it available.

Observable Regulatory API Management as a Remedy

The answers we are looking for to help us understand in an ongoing way if Facebook, or other platforms are engaging in anticompetitive practices via their platforms exists today within the API management layer. The only thing lacking is that platforms are not compelled to make the outputs of their API management layer observable by outside interests. There is a precedent in the United Kingdom for a more observable regulatory version of API management, within the banking industry The UK government mandated that all banks adopt a set of standardized APIs across common resources like ATM locators, bank account, and credit card APIs, while also participating in an API management layer that is operated by a neutral entity, which brings the following elements to the regulatory table.

API Definitions - Providing the definitions for each of the digital resources and capabilities being made available via APIs, so that which APIs are made available, as well as each version, are agreed upon in the interest of everyone involved.
Accounts - API account signup and login are managed by a neutral authority, providing observability into who all of the actors are, as well as who onboards or is terminated as part of the regular goings on across the API ecosystem.
Authentication - The authentication by developers with APIs is handled using a common set of industry standards as defined by the neutral entity, ensuring that all access is secured using common standards, and consistently applied across all APIs being regulated.
Discovery - A directory of all API consumers is available, making a subset, as well as summary data available publicly, but also providing discoverability to regulators and other government stakeholders when it comes to defining the landscape of an API community.
Management - The API management layer is standardized and externalized as part of the regulation of APIs. Each API provider, banks in this case, are required to implement the management layer as part of their operations, but the definition and oversight of what is API management is agreed upon and reported upon by the central body.
Dispute Resolution - When API access is terminated, or other challenges arise within the API ecosystems, the central body provides dispute resolution between the provider and consumer, helping bring observability but also standardization and fairness to the discussion around termination or continuing the access to API resources by applications.

API management provides us with many of the controls we need to put in place to help regulate the API access to digital resources and capabilities--it is how leading platforms have gotten where they are. We also have a precedent for how this ubiquitous approach to managing APIs can be externalized and run in a neutral way through a public and private sector partnership. We have the makings for a pretty straightforward approach for helping prevent Facebook from engaging in the same anticompetitive behavior they have been engaged in for the last decade--we just need the will to properly begin regulating the tech industry using their own infrastructure.

Adding a Government Access Plan to API Management

API management service providers deliver their API provider customers with a dashboard, and ironically, API access to this layer of their infrastructure. Meaning there is an API for accessing the logs behind who is accessing APIs. There is no reason that a regulatory access tier could be added to the API management layer by default, providing a summary of what is going on across a platform. This can be done in a way that protects the intellectual property of a platform like Facebook, as well as the privacy of end-users. It can be done in a way that doesn’t impact the performance of applications, or introduces friction into the already challenging work API developers encounter each day. All of the answers the FTC seeks to understand if Facebook is continuing to engage in anticompetitive behavior already exists, we just lack the API management plan to give them the access they need.

Adding a Researcher Access Plan to API Management

Facebook is going to continue to innovate, move fast, and break things when it comes to their platform, and API ecosystem. The company has made it clear they have very little interest in changing their behavior, and have only gotten better at apologizing and obfuscating along the way to stay off the radar of regulators. Even if regulators did impose a more observable approach to API management, Facebook is going to continue to innovate in how they reward partners and punish competitors when it comes to doing business on their platform. The pace and speed at which Facebook and other platforms move it will be also necessary to mandate a research plan at the API management layer where approved researchers can gain access to platform wide digital resources access and usage at the API management layer. Allowing researchers from universities and other institutions to come in and help quantify and make sense of what is happening in real time, helping bring operations into focus, while also helping regulators keep pace with Facebook and their strategy. Providing a secure, observable way to make sense of what is going on in real-time, helping minimize future abuse.

All Access to Digital Resources and Capabilities Go Through APIs

The first thing any tech person in the know will say to everything I have laid out is that Facebook will just route access to digital resources and capabilities outside the view of API management. Providing direct database and system access to trust partners, applications, integrations, and ensuring that only above the board activity shows up with the API management layer. Fair enough. Then Facebook should be required to route ALL external access to their digital resources and capabilities through their APIs. APIs are already the preferred approach of platforms when it comes to moving digital bits around, and making them available to partners--there is no technical or business argument for stating that APIs can’t be used. Platforms will make claims that APIs will slow things down, but there are plenty of high speed solutions available in the API toolbox, and API management can be done without injecting latency into API activity at any scale. Again, we have what we need to solve this problem, we just need the will to mandate that observably managed APIs become the normal mode of doing business today.

Solving Platform Illnesses Beyond Just Anticompetitive Practices

While outside the scope of this FTC filing, the adoption of regulatory practices at the API management layer would have wider implications when it comes to addressing some of the illnesses that plague our online world today. Having a more observable API management layer would help address a couple of the top concerns we have when it comes to depending on platforms like Facebook.

Data Exploitation - An observable, regulated API management layer would help reveal the type data exploitation of end users as we saw in the Cambridge Analytic scandal. To conduct it’s questionnaires and obtain access to the data of Facebook users, Cambridge Analytica had to sign up for API access, and provide a reason for their application. There is also an accounting of the number of OAuth tokens their applications would accrue every time a Facebook user provided access to Cambridge Analytica. The problem is Facebook wasn’t interested in punishing this type of behavior in the moment, until it became a problem for them. With a more observable API manamange layer, regulators and researchers could help identify these problems as they emerge, as opposed to willfully being blind as Facebook was because they were more concerned with their anticompetitive strategy, than they were with protecting end users.
Breaches - If all digital resources are routed through APIs, and all API access is managed in a standard and observable way, the chances of a breach will be much lower. Breaches often occur via custom, unmaintained, and legacy infrastructure. If all applications and integrations are forced to go through a standardized API layer, possessing a standardized API management layer, situations like the Cambridge Analytica abuse which technically wasn’t a breach (but it was), and other malicious activity can be identified and addressed, even if it isn’t a priority of the platform provider.

Modern API management infrastructure has been proven to help standardize and streamline how data, content, media, algorithms, and infrastructure can be securely made available online. APIs are behind every major shift of the last twenty years, and are at the heart of how Facebook rose to dominance, and executed their anticompetitive strategy. Leveraging these approaches to managing APIs would help in ensuring personal social networking, photos, messaging, and every other area of our online lives that Facebook touches enjoy healthy competition--which we all can agree will be better for everyone involved.

Now Is the Time for Sensible, Observable Regulation of Technology Using APIs

I know that I am going to upset many people I know in the tech industry with this post. It is unforgivable in many libertarian corners of the tech sector to call for regulation of our industry. I disagree. Now is the time. If you look at other transformative industries like electricity, telecommunications, and transportation, which all were heavily technology driven, regulations have played an important role in not just taming and controlling industry, but also made them safer and more accessible for everyone. More importantly, it laid a more solid foundation for these industries to grow way beyond what their early entrepreneurs or investors envisioned. I am guessing that all of these people on the ground floors of these industries claimed the same thing in the early days of regulation of electricity, phones, and automobiles. Wildly missing the scope of how these technological advances impact our lives, and the role that government would play in actually making these the world changing industries they are today.

I am writing this post, not because I believe in regulation, or hate Facebook. I am writing this because I believe in the potential of APIs when they are done in a balanced and observable way. I have witnessed APIs invade and alter almost every aspect of our now online, but also our offline lives over the last twenty years. I could see the potential of APIs when it came to commerce, social, cloud, mobile, and other major recent shifts in how our world works by 2010. But, by 2015 I was witnessing some pretty alarming behavior by leading API providers like Facebook, Twitter, Google, and others. To the point where I am pretty embarrassed by helping be a spokesperson of this industry. I am not trying to stick it to the tech sector with this suggestion that we use APIs and API management to regulate things. I am trying to take back the positive outcomes of using APIs from the greedy clutches of a few short sighted technology platforms. Ultimately I am convinced that there is even more money to be made, and even more innovation to occur, if we get down to the business of installing sensible regulation for the technology sector, and in my opinion, APIs are the key to getting it done. I am not under any delusion that this will be a quick fix for all that ails us, and I am pretty sure that initial legislation will not be sufficient. Not because we don’t have the right tools in our toolbox, but because of the lack of will and understanding among politicians, and the control the tech sector has over our government. However, this won’t stop me from trying, and I am guessing, based upon my experience so far in working with folks in DC, that I will be writing many more of these posts over the years, and working with waves of lawmakers wishing to make any change, until we reach a point where we can balance things back out a little bit, and wrestle back control from the tech platforms who dominate our lives today.

Gathering My Thoughts On Public Api Workspaces

Sat, 05 Dec 2020 00:00:00 +0000

published: true

title: ‘Gathering My Thoughts on Public API Workspaces’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/bf-skinner-thinking-man-statue.jpg

I have been neck deep in the release of Postman public workspaces lately. So much so, I haven't had much time to gather my thoughts about what exactly they are, and why they matter. Time for a burst of storytelling to help me make sense of just what is a public APIi workspace. One of the most common responses I've heard from folks that I talk with about public workspaces is that they are most comparable to a public GitHub repository, but designed just for APIs. Fair enough. While I see Postman public workspaces as much more than a Github public repository, it provides a great place to start when it comes to helping folks understand the potential. To help me be more articulate when it comes to speaking about why public workspaces matter, let me compare them to public repositories, and speak of the impact that GitHub has had on my reality, as well as on the wider API community.

GitHub Changed My Life

I have been using Github since 2010, but beginning in 2014 I began operating a significant portion of my API operations via GitHub. I ran all of my public presence there from 2014 through 2020, and I still regularly use it as a base for many API projects, and for publishing JSON and YAML files. In the last year I have pulled back much of my web presence for API Evangelist back from Github, but I still depend on it for most of my projects for the following reasons.

Free - It is free to publish repos to Github, making it a pretty sweet place to publish APIs, schema, JSON, YAML, code, and other artifacts.
Network - The network effect of Github is huge. Developers get it, and the access and discoverability that comes with the platform are worth it.
Social - The social layer GitHub added to Git is really what makes the platform as powerful as it is, making things more collaborative by default.
Source = Github is designed to be the source for code, but it works just as well for OpenAPIs, JSON Schema, and other artifacts you are managing.
README - The readme for each repository provides a great hello world for each idea and project, allowing everything to be born as a repo.
Issues - I use Github issues for everything! It is how I aggregate tasks, features, bugs, and engage with people via the hundreds of repos ii have.
Pages - Github Pages combined with Jekyll has transformed how I publish data-driven websites, dashboards, and tools for others to use.
SEO - Early on GitHub repos didn't index well, but eventually it became better to publish things to Github than it was to your own properties.

Github changed my life. Github is how I built API Evangelist. Github is how I made sense of the API lifecycle. All my ideas for the last five years have been born and raised as a Github repository. Repositories provided me with a standardized, yet extensible way to define and move forward an API. A repo, combined with Github Pages, Jekyll, and powered by the Github API gave me a robust platform that provided me with about 60% of what I needed to manage my API operations. Github was the factory floor of my API operation, as well as the public performance of those operations. It is not an understatement to say that Github changed my life—it did.

Comparing Postman Public Workspaces to GitHub Public Repositories

Github gave me a pretty extensible base for my API operations, but I found myself always hacking at the edges of what the platform could do to actually deliver what I needed. I had a lot of custom code running on AWS that was required to automate and orchestrate the API factory floor I was operating on Github. This comparison between Postman public workspaces and Github public repositories provides a good starting point for many conversations, but will quickly fall short when it comes to actually describing what is possible. Yes, public workspaces are publicly available like public repositories, and you can store and collaborate around artifacts, but Github was designed for developing open source software and Postman is designed explicitly for APIs. Providing a much more powerful set of tools that help you define the factory floor of your API operations, with each workspace possessing the following resources.

APIs - Defining APIs using Swagger, OpenAPI, GraphQL, and RAML, providing the contracts for each of the APIs you are providing or consuming as part of operations.
Collections - Developing and generating collections of API requests which may or may not be defined by OpenAPI contracts, and used across the API lifecycle.
Environments - Defining different sets of key / value pairs that provide variables that can be applied across collections, applied to authentication and other needs.
Mock Servers - Generating mock servers from collections, providing mocks of APIs that can be used to design, develop, and testing of APIs across operations.
Monitors - Allowing for the running of collections on a schedule via different cloud regions, executing collections across the API lifecycle to automate operations.

I can store OpenAPIs, collections, and other artifacts on Github, but it doesn't provide me with the ability to execute and iterate upon the purpose of each of those artifacts. GIthub provided us with a number of things that help us change how we deliver software, by opening things up, making things more social and collaborative. There is definitely an essence of Github repos present in Postman workspaces, but we are taking what we've learned about optimizing the software development lifecycle, and community aspects of the open source world, but then applying them the next generation of the API lifecycle, which in turn will continue to define how we deliver software applications. To understand the implications of these API centered resources available within Postman public workspaces, it helps to walk through a handful of use cases for public API workspaces, and see them in action.

API Provider Public Presence Workspace - Salesforce

Like with the Postman public API network, the first place people will go when thinking of a public API workspace will be providing a place for public API providers like Salesforce to publish their APIs, collections, and engage with their consumers. While there is just a lone collection in this official Salesforce public workspace, I think it represents how we'll see most API providers begin their public API workspace journey.

One of the things our partners are dealing with in their own way is how complete, ready, or polished a workspace needs to be. With some of them realizing that it is first and foremost a workspace, which means it will be a place to do the work to complete, ready, and polish the APIs, collections, and other moving parts of a public workspace. Realizing that iit is a journey, and you are best to just get something stood up, then spend the time polishing what is happening using the feedback from your community--not opting for the perfect workspace.

API Provider Public Support Workspace - Dropbox

Dropbox has also published a public workspace under their team page, but they have applied a different level of prioritization when it comes to what they are publishing to the Postman network, but also to their workspace. In the Dropbox public workspace you'll find a single collection designed for supporting Dropbox admins, helping automate some of the most common support requests from their admin API consumers.

This reflects where I think that API providers should be headed with their API collections as well as public workspaces. Every API provider should start by publishing an OpenAPI and/or a reference collection for all of their APIs, but then quickly get to work on collections that put their APIs to work to solve real world business problems. I think the Dropbox approach demonstrates the diversity that will exist even amongst this most common definition of what a public workspace i'll be for an API provider.

API Blueprint Public Workspace - Products

When I first got the ability to publish a public workspace I immediately thought about being able to use demos, webinars, and workshops to show people how to build APIs. I have one private workspace that you will see me use in all of my demos to demonstrate an API-first lifecycle, and the potential of Postman as an API development tool--a reference implementation for a simple product API. Something I made sure to fire up as a public workspace, showing one possible way to develop APIs.

I feel like this is the type of workspace that will have the greatest impact on the API community, but I am a little biased towards workspaces being educational. Regardless, I plan on publishing a variety of these types of public workspaces to demonstrate how someone can approach the API lifecycle in a more structured and consistent way using Postman public workspaces, using OpenAPI, collections, and other key Postman platform resources.

Industry API Standards Public Workspace - UK Public Banking

One project I am working with our runtime partner APIMetrics on is showcasing how API regulation is being applied in the United Kingdom to make banks more transparent, observable, and consistent. Resulting in a UK public banking API workspace with a handful of OpenAPIs and collections for a handful of the public APIs in which the UK government mandates all banks possess.

This type of API workspace demonstrates how APIs can be standardized to help drive a specific industry. Showing how one OpenAPI can be used in conjunction with Postman collections to move forward the design of APIs, but also move them forward across the API lifecycle, and demonstrate how they can be applied across multiple API providers.

Industry API Standards Public Workspace - FHIR API Specification

Continuing to build on this type of industry public API workspace for defining an API that can be used by many API providers, I fired up a public API workspace for the Fast Healthcare Interoperability Resources (FHIR) API. With this workspace I am looking to demonstrate how a workspace can be used to move forward an industry wide standard for the healthcare space, while also showing it in action with documentation, mock servers, and suite of testing to validate the instance.

The FHIR API specification is one of the most important API specifications out there right now. You can see this API being applied as part of the Department of Veterans Affairs (VA) APIs, as well as Center for Medicare and Medicaid Blue Button APIs. LIke many of the workspaces listed here, there is a lot of work needed on this OpenAPI, it's collection(s), examples, and other moving parts of demonstrating what is possible with the API.

Municipal API Standards Public Workspace - 211

Following a similar approach to the UK banking and FHIR API public workspaces, next up is the Human Services Data API (HSDA) which is used to deliver 211 services within cities. Helping make human services across the community accessible by those who need it to deliver a variety of web and mobile applications. I am the technical lead on the HSDA API, and will be using this workspace to move forward the specification, while demonstrating what is possible with the API.

HSDA is a pretty critical API for helping ensure our communities are healthy and liveable. Ensuring that critical organizations, locations, and services are accessible by citizens within a community. Providing a common interface that can be used within a single community, but also used to harmonize web and mobile applications across many different cities.

Municipal API Standards Public Workspace - 311

Complimenting the HSDA 211 API above is the same but for 311 services, helping provide a common API that communities can use to engage with the community around the reporting and servicing of non emergency events that occur across a community. Adding to the stack of API standards that municipalities can use to effectively manage a city and all of the goings on across them each day via a 311 public workspace.

I am working with the technical lead, and other stakeholders of the 311 standard to move the specification forward, showcase the cities that have embraced the standard, and demonstrate what the potential is when you put this API standard to work. Both the 211 and 311 API standards provide us pretty solid examples of the importance that APIs and public API workspaces can play in making our lives more livable.

API Business Sector Public Workspace - Commerce

I am working my way through the thousands of APIs I have indexed and publishing and aggregating APIs into a variety of business sector public workspaces. I started with all of the commerce related APIs I have available, and publishing the OpenAPI for each one, then generating a collection for the purposes of documentation as part of a single workspace. Making it easier to browse through and kick the tires on all of these APIs, side by side in a single public workspace.

Commerce is the original reason for doing APIs, so I figured I'd start here when it comes to showcasing the potential of public workspaces. Seeing APIs side by side like this, then being able to assess the potential available across all of them is something that is going to continue to evolve how I see APIs in these sectors, but also how I see public API workspaces.

API Business Sector Public Workspace - Message

After commerce I was eager to see all of the messaging APIs side by side to better understand this sector. Aggregating all the different types of SMS, email, and other APIs into a single public workspace so that they can be used together, or just used to understand this slice of the API pie.

Messaging is one of the most relevant sectors of the API space, providing a suite of APIs that will resonate with a wide audience. Providing a public workspace that I can use to help drive conversations with developers as well as business users. Messaging comes in many shapes and sizes, and this workspaces helps bring the sector into focus for me, and I will keep aggregating, publishing, and evolving APIs to show what is possible.

API Topic or Trend Public Workspace - COVID-19

Driven by the success of our COVID-19 resource center, as soon as public workspaces were available we published the best of the collections from the resource center into a single public workspace. Showing how a single topic, trend, or other bounded context can be applied to a public workspace, making a variety of APIs available in a single place for developers and other users to put to use in their applications.

I am looking forward to adding more automation, visualizations, and other elements to this workspace. Showing how public workspaces can be used to make these types of APIs available, but can also be used to better make sense of what is happening around the world when it comes to COVID-19.

API Topic or Trend Public Workspace - US Election

After we launched the COVID-19 resource center we wanted to apply the same approach to making sense of the US election this year. Aggregating the best of the election related APIs into a single workspace where developers can fork and put to use, while also engaging with each other when it comes to getting their questions answered.

Now that the election is over I am looking forward to showing why these APIs matter during an election, but also afterwards to help us make sense of what is happening. Demonstrating the importance of APIs in each election, and that we should be investing in standards as well as individual implementations in perpetuity, not just as we ramp up for the next election.

API Automation and Orchestration Public Workspaces - OpenAPIs

I spend a lot of time answering OpenAPI related questions which are a collection. Helping demonstrate a specific aspect of working with OpenAPIs to Postman customers and the community. To help me be more organized around publishing, evolving, sharing, and collaborating around these collection driven capabilities I published a single workspace for managing my OpenAPI collections, orchestrations, and automation.

As the number of my workspaces increases the importance of having these types of collections organized in a single workspace becomes even more important. I normally would just organize these across a variety of personal and team workspaces, then share them as needed, occasionally publishing to the API network as a template, but now that I have a public workspace I am being a little more organized about publishing them to a single place and then linking to them via a single URL.

API Automation and Orchestration Public Workspaces - Collections

I spend a lot of time answering collection related questions. Helping demonstrate a specific aspect of working with collections to Postman customers and the community. To help me be more organized around publishing, evolving, sharing, and collaborating around these collection driven capabilities I published a single workspace for managing my "collection collections", orchestrations, and automation.

Government API Public Workspaces - Federal Election Commission (FEC)

One area of the API economy I am keenly interested in is when it comes to government, and after publishing the FEC API to the US election public workspace, I figured I'd add it to a workspace for the agency. Providing a single workspace where you can find all of the APIs from this very critical federal agency.

For me, a workspace provides a reference to each government agency, helping remind me to invest some time in researching what is happening with APIs at this government agency. Allowing the workspace to drive work around the government agency, helping keep things up to date and representing what is going on with APIs and open data in government.

Government API Public Workspaces - Department of Veterans Affairs (VA)

Next up to demonstrate the potential of APIs in government is out of one of the government agencies that is near and dear to my heart, the Department of Veterans Affairs (VA). I took all of the OpenAPIs the VA provides as part of their developer area and imported them into a single workspace, generating collections from them to help document what is available.

The VA Lighthouse effort is one of the solid examples of APIs making a difference in government, and I am keen on demonstrating how public API workspaces can be used to help the VA better serve veterans. Providing a pretty compelling use case for public workspaces that can help make a difference in the lives of Americans.

Just the Tip of the Iceberg of What is Possible

It took me building out different types of public workspaces before I began understanding what a public workspace is. Honestly, I don't think I've even touched on 10% of the possibilities with these examples. Like GitHub, it will take me several years, and building out hundreds or thousands of public workspaces before I'll have a handle on what is possible. Honestly, I am hoping that the community will show me what is possible, as I don't always trust my imagination when it comes to mapping to the reality on the ground across enterprise organizations, institutions, and government agencies. I am betting on the innovation and ingenuity of developers to show what is possible within a public workspace when it comes to helping them better achieve their objectives in making APIs available to a public audience. As usual, I am just hoping that my storytelling might just light the fire under the imagination of folks with these examples.

Bringing Collaboration To API Operations

Collaboration is one area public workspaces reflect a public repositories. It is definitely one of those areas that the open source movement has impacted how we engage across the API lifecycle. However, the public workspace is much more API centered, focusing on the API contracts and the collections that deliver and validate these contracts throughout their lifecycle. Here are a few of the collaborative aspects of public workspaces, that contribute to bridging the gap between provider and consumer.

API Watching - Being able to watch APIs available in the public API network, and receive updates when it changes.
API Commenting - Being able to comment on APIs, and the parts of APIs, establishing a feedback loop around APIs.
Collection Commenting - Being able to comment on collections, and the parts of collections, establishing a feedback loop around collections.
Team Member Mentions - Being able to mention team members in API or collection comments, allowing for engaging with specific team members around APIs and collections.
Forking - Allowing team members and the public to be able to fork a collection to their own private or team workspace, and put to use, or make changes to their forked copy.
Pull Requests - Submit any changes made to a collection back to the collection owner, allowing them to potentially accept and merge changes back in.
Merging - Allowing the owner of a collection to accept pull requests from team members or public users, merging any changes back into a collection.
Sharing - Encouraging team members to share APIs and collections to workspaces where they will be needed, or share the URL of a collection to others via email or other messaging.

These collaborative features are available within each workspace, rolling up to the activity feed for team members across workspaces. Making the API lifecycle a much more collaborative process, allowing team members and the public to tune into what is happening around APIs, and then engaging with the process throughout the API lifecycle. Bringing API development out of silos and helping make it much easier for developers and other API stakeholders to work in concert, and stay in tune with what is going on across an organization.

Automating the API Factory Floor with Public Workspaces

CI/CD have become synonymous with software delivery lifecycle, and while the deployment of APIs has benefited from this evolution, we are going to need much more automation, orchestration, and establishment of repeatable processes as part of the API lifecycle if we are going to continue to reliably deliver the API infrastructure at the scale we'll need for the next generation of applications. Postman workspaces provide three distinct approaches to automating the collections that are defined and available via the API workbench that exists within a private, team, or public workspace.

Runners - Manually running a collection and all of the requests it contains in a single motion, triggered by a Postman team member.
Monitors - Scheduling the run of a collection on a specific schedule and from different cloud regions, automating any possible collection.
Pipelines - Executing a collection as part of a pipeline weaving API automation into the existing automation used in the software delivery lifecycle.

Think of a public workspace as that microbrewery restaurant that has the view into how the beer you are drinking is actually brewed. Letting everyone see the entire process from start to finish, instilling confidence in consumers that they are getting the best possible beer, made from the best possible ingredients, using the most streamlined process possible--it is so good, we want to show you all how it works.

Observability By Default with Public Workspaces

In addition to the collaborative possibilities of public workspaces, they inject observability into the conversation by default. All workspaces have outputs that can be used to understand what is going on in real time across operations. Allowing API operations to be understood by capturing the exhaust from regular activity, then using it to stay in tune with what is going on, and respond to changing conditions. Introducing more visibility and accountability into the API lifecycle, without making people do anything extra, other than just getting their regular work down around providing and consuming APIs--here are the elements of a public workspace that contribute to observability out of the gate.

Workspaces - Just the fact of having a workspace provides a place to find out what is going on within a single bounded context.
History - Showing all of the requests made by collections across team members, recording every API call made, and allow it to be filtered and used.
Activity - Having real time visibility into all the activity from across all elements of a workspace, providing true API observability using the Postman platform outputs.
Reporting - Providing default reporting all the elements within a workspace, providing a visual dashboard for what is going on in real time across teams.
Visualizer - Using the visualizer tab for each request, making API calls to better understand what is going on with each API, or across APIs, and rendering using a custom JavaScript library.
Comments - If conversations are localized to workspaces, and made even more precise around the parts of each API and collection defining the API lifecycle, you end up with a powerful feedback loop that makes operations more observable by all.

Being able to understand API operations using the outputs of your API operations is essential to developing an awareness of what is going on at scale. APIs are abstract and difficult to see. When you have hundreds or thousands of APIs, the challenge becomes even greater. API observability begins with monitoring the APIs themselves, and the infrastructure behind them, but then observability should mature to be also about what teams are up to, and being able to observe, guide, and respond to what is happening across operations in real time.

Multiple Dimensions of API Collaboration

One of the things that stands out for me across these example workspaces I've created is that there are multiple levels of collaboration, depending on what you are looking to accomplish with your public API workspace. I am sure there are more dimensions than what I have fleshed out here, but these three dimensions of collaboration that really stand out for me as how we optimize operations to work for eveyrone involved.

Provider - API providers can use workspaces to better define the different parts of their operations, being more public in their operations when possible--walking the walk when it comes to operating the API infrastructure in a way that you want to showcase to others.
Consumer - You don't have to be an API provider to operate a public workspace, and as shown above you can aggregate, automate, and orchestration using many different workspaces via a dedicated public workspace.
Standards - Public workspaces are great for moving forward API standards, helping ensure there is the necessary collaboration in place to move a specification forward, while also demonstrating how an API works to providers and consumers.

You can accomplish this type of collaboration via Github repositories, but you just won't find this level attention to the API details in a repository. Github is designed for an open source software development lifecycle. Postman public workspaces have been defined by both API providers and consumers over the last 5+ years, resulting in a workspace approach that already works for industry, government, and other standards bodies looking to guide both API providers and consumers to all move within the same direction--using a tool that is already ubiquitous across development teams.

API Discoverability Baked Into API Operations

Public workspaces makes your APIs, collections, and the working occurring around them more discoverable. The new Postman search takes discoverability to an entirely new level all by itself, but operating your APIs in a public way adds a whole new level of discoverability. The ability for APIs to open up enterprise organization and push teams to evolve beyond their internal siloes is one of the reasons I started studying APIs in the first place. There are benefits of letting the sunlight into our operations, and operating our API infrastructure out in the open. Sure, not all APIs should operate this way, but most will dramatically benefit from being exposed to the watchful eye of the community it serves. Making the workspace in which APIs, collections, documentation, testing, mock servers, monitors, and other moving parts of our operations public by default helps ensure consumers can find APIs in the first place, but then it also goes a long to build and maintain trust with consumers without doing much more than being more discoverable, accessible, and transparent in everything you do.

The Versatility of Collections and Environments

There are so many possibilities of what a public API workspace can be because of the unlimited possibilities in how a collection can be defined. Anything an API can do, a collection can do. Once you start daisy chaining multiple APIs together within a collection the possibilities become pretty dizzying. Transcending what is possible with any single API resource or provider and really showing the potential of a distributed API economy. A collection isn't like an OpenAPI in that it tends to represent a single set of API resources or capabilities, it is something that can reflect a series of API calls that reflect a set of business objectives. Providing a unit of valye that can be applied one time, on a schedule, or as part of the regular software development lifecycle. Once you begin to establish abd evikve a=tiyr workbench for crafting, evolving, collaborating and orchestrating out in the open with your community around APIs, and collections around them, the possibilities really are limitless.

Pulling Back the Curtain of Our API Operations

Public API workspaces have the potential to pull back the curtain of our API operations. I know that this makes some folks nervous. If your APIs are being consumed by 3rd party applications and integrations you should be working to be as public with your efforts as you can. The least amount of distance between you and your consumers is optimal for putting APIs to work, as well as evolving them. Having a feedback loop in place around each of your APIs, but also the moving parts of your API operations like documentation, mock servers, testing, and other elements helps grease the wheels of your operations. Supporting API consumers in this way will be foreign for some API providers, but as I said before, it will let in some much needed sunlight on your operations, bring your API consumers closer, and give them a voice in the design and delivery of your API infrastructure. The distance between API consumer and producer, as well as the lack of a two way feedback loop is the top deficiencies that plague API platforms tidat. Public API workspaces help pull back the curtain on your API operations, helping bring in the communication that is required to optimize your API factory floor, as well as the supply chain betweeb you and your consumers.

Emulating the Best API Patterns Across the Landscape

After playing this API game for over a decade I notice that most people just emulate what they see. If people have consumed a number of different APIs, and are exposed to a wide variety of conversations around APIs, they tend to provide APIs, and run their operations similar to what they've seen. If people haven't consumed a number of different APIs, and have been exposed to a variety of conversations, they tend to not understand what makes APIs work. The more isolated API providers are in their API journey, the more friction and instability will be present across API operations. APIs are all about using the web to make digital resources and capabilities available to the widest possible audience, while striking a balance between being publicly accessible, but secured and generating revenue, ensuring APIs remain sustainable for everyone involved. Being comfortable with operating your APIs out in the open takes a certain amount of confidence and expertise, but it is what helps an API ecosystem thrive. It allows all of us to learn from each other. I help us share the healthiest patterns for operating our APIs. Not just telling people how they should operate their APIs, but demonstrating how to do it. Then API providers can decide which approach suits their needs, and will help them best serve their consumers. Allowing us all to emulate the best of what we are exposed to, pushing us all to be a little better in our own operations because we know that our consumers, and the wider community are watching.

Using Workspaces to Define Your API Narrative

A public API workspace is just that, a workspace. It is a place to work on your APIs. A public workbench is for defining the contracts for your API, as well as the collections you need to document, mock, and test your APIs. Then the environments, runners, and monitors you need to move each API forward from design to production, while also providing you with what you need to manage change across this API factory floor. How big or small your workspaces are is up to you. Just exactly what goes into a workspace and how it operates is up to your team. You can have as many or as few workspaces as you need to get the work you need done around providing and consuming the APIs you depend on. APIs are a journey, and not a destination. You will never be done delivering APIs. You will never be done with needing to discover and use 3rd party and partner APIs.You will never be done with using APIs if you are going to do business in a digital world. Public API workspaces reveal that this work is never done, and that we will need to setup and maintain private, team, and public workspaces for defining all the digital resources and capabilities we need to do business each day.

I am very accustomed to operating in a very public way. Not because I am in search of page views, more followers, or user signups. I am public in this way because I know that I learn more from this type of production, and that others in the community will learn from it as well. I depend on the feedback loop it creates to find new people, work, ideas, and energy to keep going. Github was one the top platforms from the last decade which have transformed the way I do business, and influenced how I see and use APIs. I am fully aware that I am biased, but I see Postman public workspaces doing this as well. Public workspaces have already changed how I do my storytelling. You'll notice many of the blog posts I am writing now contain a link to one or more APIs or collections in a Postman workspace. You'll see that some of the overview pages for my public workspaces contain todo lists of collections I want to build, as well as the stories I want to tell. Public workspaces are making my storytelling more hands-on, interactive, and even forkable. I see this as being transformative in not just how I support my community, but how API providers and API service providers support their communities as well--seismically shifting how we engage with each other in this API-driven world.

Expanding The Vocabulary For Run In Postman Buttons

Sat, 05 Dec 2020 00:00:00 +0000

published: true

title: ‘Expanding the Vocabulary for Run in Postman Buttons’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/screen_shot_2020_12_05_at_4.20.41_pm.png

I have long been fascinated by the Run in Postman Button. A Run in Postman button can be published from any Postman collection, organizing a single, or a series of API calls into collection of API requests, and then letting them be imported and ran locally by any consumer, as a cloud monitor, or via a pipeline. Run in Postman buttons are a common way for API providers to onboard new developers with their APIs, and stay engaged with active developers through collections they’ve download via Run in Postman buttons. These embeddable goodies attached to Postman collections and potentially environments are very powerful unit of API execution, but I am finding that the label “run” isn’t sufficient in articulating what is possible with each collection, and I a looking to get a little more precise when it comes to my vocabulary attached to each Postman button I publish.

The Run in Postman Button Fundamentals

A Postman collection is basically a portable folder of API requests. Using Postman, developers can define the parts of one of many API requests, including the URL, parameters, headers, body, and authentication, then roll them all up as a collection that can be shared with other developers, and published as documentation, as well as using a Run in Postman button. This embeddable HTML or markdown button can be published alongside documentation, or more dynamically as individual buttons on any HTML or markdown page, attaching each button to the following elements:

Collections - Each button can execute a single collection of one or many API requests, complete with pre-request and post-request scripts.
Environments - Each button can possess a single environment that can contain a variety of key / value pairs for authentication, and other purposes.

You can find Run in Postman buttons sprinkled across Twitter API documentation, providing one click import of collections into each developers own workspace, where they can manually run, or automate using a monitors, or trigger via a pipeline using the command line runner. This powerful combination allows for an almost unlimited number of possibilities when it comes to crafting a collection, environment, and then publishing using a “Run in Postman” button—something “run In” just does not adequately represent when articulating what is really possible.

Exposing the Semantics of Each Collection

The label “run in” provides a powerful, yet generic wrapper for each collection. I’d like to see a semantic layer emerge for Run in Postman Buttons, allowing me to better define and articulate what is happening within each collection. Yes, each collection is being “ran”, but I want to be able to get more precise about what is happening within each collection, moving beyond the notion of a “reference collection”, or just a buffet menu of what is possible with an API, to very precise capabilities that are enabled by an API. Here are a few labels I would use for some of the collections I currently use across my work.

Deploy Using Postman - I have a variety of collections that will take a fresh OpenAPI and publish it to AWS API Gateway, Lambda, RDS and / or DynamoDB.
Clean Up Using Postman - If an API I have deployed is ephemeral I will have a second collection that I can use to tear down and clean up an API deployment.
Harvest Using Postman - I have collections that will harvest data for me from Twitter, GitHub, and a myriad of other APIs I am regularly pulling data from.
Sync using Postman - I have collections that I use to sync data between two platforms using APIs, running one time or on a schedule to migrate data.
Publish Using Postman - I have many static sites that run on GitHub using GitHub Ages, and I schedule the publishing of JSON or YAML data via a collection.
Scan Using Postman - There are a collection I use to scan the surface area of APIs looking for vulnerabilities and other security issues across my APIs.
Test Using Postman - I have many different types of tests that live as Postman collections, allowing myself, or others to run tests manually or on schedule.

This is just the tip of the iceberg when it comes to Postman collection defined API capabilities I need to run, share, and publish with other people. I am regularly answering questions around how to do something with APIs, OpenAPIs, JSON Schema, and other API lifecycle challenges, helping provide a solution as a single collection. I have diverse vocabulary when it comes to how I run my collections and it would be nice to be able to expose the semantics of my APIOps as simple buttons that I could publish in documentation or as part of the storytelling here on my blog.

The Automation of API Capabilities Using Collections

While the semantic of each collection and its underlying requests ands scripts would be better exposed via a semantic label on a button, each API capability could be executed in a variety of ways using existing Postman platform functionality. Starting with the ability to discover and learn about a specific API capability via documentation or wherever buttons are published, but then quickly move beyond manual actions, helping automate all of these capabilities via these methods:

Single Run - Allowing a user to run a collection combined with an environment a single time upon clicking the button.
Scheduled - Enable a user to schedule by the minute, hour, day, week, or other dimension, baking schedule into button.
Pipeline - Embedding the capability into an existing pipeline, ensure that the capability is run in the moment it is needed.
Events - Triggering the run of a capability when a specific event occurs, leveraging a web hook to execute the collection.

I would have to explore how the vocabulary of a button label could be expanded beyond just the semantics of what the collection can do, and consider how the variety of approaches to executing an API capability via a collection might be expressed. Schedules would need to be defined, pipelines indexed and made extensible, and events registered for triggering of web hooks that set each collection + environment into motion. Ultimately I am looking for an embeddable link or button that has a label that describes the entirety of what is possible with each collection, and the method of execution, helping provide a suite of single click capabilities that developers and non-developers can use to automate the world around them.

The Observability of API Automation and Orchestration

Right now I can tell you the number of times a collection was viewed or downloaded for any collection in the Postman network. This provides a reliable way for API Providers to track the onboarding of new users, and since they have API management in place, they can link this onboarding to API activity by each user onboarded. Providing two of the classic metrics for any API platform--new and active users. However, I can’t help API providers confidently implement, measure, and report upon what actually is activating each API consumer. Adding a semantic layer to each collection, exposed via it’s published button, provides me with a more precise dimension to report upon when it comes to quantifying how users move to differnt levels of ctivation, and different levels of activation, based upon the type of collection being employed. Giving me with more tools to help API providers experiment and quantify how to best reach API consumers.

Capability Collections - API Providers can develop precise collections that speak to a specific business value their API enables.
Service Composition - Each collection can contain one or many individual API requests that map to the API management layer.
Collection Reporting - With a semantic layer added to each collection, my views, download, and execution means much more.
Experience Journeys - As an API provider I can craft many different collections that enable a variety of journeys for API consumers.
Operational Landscape - I’ve added another semantic layer to the reporting for my APII operations, that better define my API landscape.

I really haven’t done anything groundbreaking here. All I am doing is adding a semantic layer to each collection that helps them convey more meaning to both API provider and consumer. It helps me move beyond just the buffet menu nature of existing reference collections which show API consumers everything that is possible, and then expects them to just go figure it all out. It helps be get more structured in how I am defining the capabilities of my API platform, and share, publish, and articulate those capabilities to consumers. While also letting me better report upon, develop an awareness, and evolve my API strategy to better meet the needs of my consumers, and of course my platform.

Expanding My API Vocabulary and Then Putting at My Fingertips

This notion of expanding the Run I Postman button vocabulary has been kicking around in my head for a couple of years now, but it was until we launched our new search recently that I could see the possibilities. How I name my collections matters a lot more now because of the expanded search available in Postman now. Being more thoughtful about how I name collections to ensure maximum discoverability, triggers my desire to help API providers better articulate and communicate the value that is contained within each Postman collection. Increasingly I am generating many derivative collections from an OpenAPI definition, looking to create more tangible and usable executable units of API value that speak to a wide variety of developers, but more importantly non-developers. The label on a button might seem insignificant, but really with each button you have a finite, but powerful surface for articulating the endless possibilities that exist within each Postman collection it is attached to, but also the value generated from the APIs defined as part of that collection.

The release of public workspaces has let in some sunlight on my chaotic and disorganized approach to creating collections for a variety of purposes. Publishing documentation and run in Postman buttons has always been the public layer of this chaotic performance of API Evangelist, but also now the Chief Evangelist at Postman. Exposing my wealth of APIs and collections via workspaces, then trying to use the new Postman search to find what I need, and recall the many different API-driven capabilities I have laying around has really shown me the importance of being able to properly label each collection in a way that is meaningful to me, but also to my audience. Next, I am going to continue working on how I design, develop, and label my collections, while also putting more thought into how I can publish as buttons, but maybe add my own custom layer which exposes the value contained within each collection “run”. Really, you could launch an entire button, as well as execution layer using the Postman API, managing collections and environments there, while doing the execution using monitors. I’ll spend some time tinkering, and see what I can come up with. Who knows maybe I will be able to influence the road map with this work. Or better yet, maybe you’ll build this, tell me about it, and I’ll help you influence the road map. ;-). I have too much work as it is. ;-)

A Postman Collection For Updating A Collection Host Path Or Query Parameter

Mon, 30 Nov 2020 00:00:00 +0000

published: true

title: ‘A Postman Collection For Updating a Collection Host, Path, or Query Parameter’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/screen_shot_2020_11_30_at_5.23.57_pm.png

Building on my base collection for pulling and updating collections, I added five specific requests that will help me conduct a find and replace on each API request host, path, and the query parameter key, value, and description. There are other dimensions I am looking to cover with future requests, but this gets me what I need right now. You simply add a value to the request for a find and replace value, make sure you’ve entered a collection ID (pulled from URL or info tab), and hit run—it will conduct the appropriate find and replace, and then save the collection using the Postman API. So you can immediately put to use the same collection with the changes you desired.

Helping The Public Data Commons Drive Our Economy Using Apis

Fri, 27 Nov 2020 00:00:00 +0000

published: true

title: ‘Helping the Public Data Commons Drive Our Economy Using APIs’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/bf-skinner-adam-smith-edinburgh.jpg

What is an API?

An API is a digital interface for sharing data, content, and algorithms with web, mobile, and device applications using the Internet, building on web technologies to make digital resources available across many different applications. APIs have been around since computers and their networks first gained a foothold back in the 1960s, but with the rise of the web since 2000, a new breed of APIs have emerged which has changed how we build and use technology, and introduced entirely new ways of doing business, but sadly, they have also introduced entirely new ways of exploiting an destabilizing the physical world around us. APIs aren’t the latest techno solutionism, although they are oftentimes billed as that, they are the digital currents that flow around us each day. APIs power the web and mobile applications we depend on each day, while also steadily working to redefine our physical worlds by connecting everything to the Internet—reshaping our virtual and physical worlds, while also remaking who we are as humans along the way.

Websites Are for Humans

The web as we know it has been evolving for over 50 years, but in the 1990s it became something we were able to access in our homes and businesses, laying the foundation for the ubiquitous web of applications we now access in our homes via laptops, televisions, and other appliances, as well as our cars, in our businesses, and across our communities in the form of security cameras, traffic infrastructure, digital signage and much more. Websites are hypertext markup language (HTML) that are designed to be rendered for humans in a browser, making the text, images, and other media consumable by humans using their eyes and ears, and navigated using our fingers via touch screens, keyboards, and increasingly voice controls. Websites provide us with a user interface that each person can put the web to use in their personal or professional worlds, connecting us with the digital world that has emerged before us, and increasingly throughout our physical world as well.

APIs are for All Other Applications

As the web has expanded over the last twenty years beyond simple web pages accessed via desktop computers, the builders of the web realized they needed to develop ways in which data, content, and algorithms could be reused across many different web and mobile devices. To accomplish this, developers began adapting web technologies using HTTP to make data, content, and algorithms available in much more standardized and machine readable ways so that the data and content could be used across many destinations. Separating the user interface from the data, content, media, or algorithms, widening how and where the digital resources could be applied, using them to power their primary web, mobile, and device applications, but also making the increasingly valuable digital resources available to partners and 3rd party developers, generating entirely new revenue streams from data, content, media, and algorithms which had limited value before. Resulting in a layer of application programming interfaces (APIs) on top of the existing low cost web infrastructure companies were already using to deliver their existing web applications.

History of API - Commerce

The early web API pioneers were all interested in expanding the reach of their commerce networks, making their products and services available to affiliates, resellers, partners, and 3rd party developers looking to build the next killer application. Salesforce, eBay, and Amazon were the first technology companies who began investing in API infrastructure in the early 2000s, setting into motion a whole new way to make digital resources available via the web. Translating the sale of physical products and services from our physical worlds into the online world were the most straightforward and widely understood way to apply the web to reach new customers, but it also quickly translated to more distributed approaches to selling products and services through many different applications and digital networks via simple web APIs. By 2005 a significant portion of Salesforce, eBay, and Amazon’s revenues were beginning to flow through their APIs, laying the commercial foundation for what many folks like to call the API economy, but many who were tuned into this shift would soon realize there were some other critical ingredients needed to get us where we are today.

History of API - Social

As the commerce phase of the API evolution was being set into motion another aspect of the API economy was taking shape, leveraging APIs to not just sell products and services, but to further define how humans use the web to stay connected and get their information, showing the social value of APIs. In 2003 the image sharing platform Flickr realized they could rapidly expand the reach of their platform to the growing blogosphere by offering 3rd party developers API access to their increasingly valuable media platform—realizing this was much more than just a technical advancement, but also a business one they dubbed business development 2.0. Shortly after Flickr realized the potential of APIs the world saw both Twitter and Facebook introduce two API-driven seismic shifts that we are still working to understand the impacts of in 2020. Much of the growth of Twitter and Facebook over the last fifteen years has been built on top of their public APIs, providing a self-service way for partners and 3rd party developers to extend the reach of the network, helping build buttons, badges, widgets web, mobile, bots, and device applications that would extend, augment, and strengthen the importance of these social networks, and as we’ve seen in the last decade also the automation, orchestration, and introduction of bots who continue to define the tone and balance of these online platforms.

History of API - Cloud

While the social phase of the API evolution was ramping up, one of the early API pioneers forever changed how we’d deploy our software infrastructure using APIs, with the introduction of what we now call “the cloud” from Amazon Web Services. After finding success with APIs to support their commercial ambitions Amazon had internalized APIs in a way that would demonstrate how you could use web APIs to do more than sell products, blog, and share posts with your friends on a social network. With the introduction of AWS S3 and EC2 in 2006, Amazon showed us that you could also deploy global infrastructure using APIs, opening the floodgates of the types of digital resources you could make available through web APIs. It wasn’t just data, content, and media anymore. You could use web APIs to deploy servers, store previously unheard of amounts of data, as well as configure the DNS and network for your applications in real time. This API-driven cloud gave us the modular resources we would need to build applications for the future, but it also gave us the scale and scope we needed to deliver the distributed applications we would need to do business in the digital age. Adding another important dimension to the API conversation that would make the next seismic shift possible, moving the Internet from servers in data centers into the cloud, and then into our pockets with mobile applications.

History of API - Mobile

With the introduction of the iPhone in 2007 and the Android mobile phone shortly after, a whole new world of applications emerged. Moving applications from our desktops and laptops into our hands, mobilizing common applications and platforms we used so that we could take them with us, expanding the reach of the web to wherever we could get a cellular signal. Developers quickly realize that they could deliver the same data, content, media, and algorithms they used in their web applications to the new mobile applications they were delivering. Realizing that they could deliver digital resources once using APIs then reuse over and over in many different web and mobile applications, while also making them available for use by trusted partners, and 3rd party developers when it m made sense. Using web APIs to deliver the digital resources needed to power mobile applications allowed application providers to use the same low cost web infrastructure they used for their websites to deliver mobile applications, helping reduce overhead and become more agile and nimble when it comes to delivering new applications, but also iterate upon each version of existing applications. The table for the API economy was being set, and APIs were becoming the preferred approach developers needed to deliver the digital resources used in web and mobile applications, but soon would expand to anything that could be connected to the Internet, pushing APIs beyond the digital world, lighting up common everyday physical objects with the ability to read and write data via wireless networks.

History of API - Devices

APIs demonstrated that data could provide a rich experience via mobile devices, and it didn’t take too long for developers to push these capabilities to a new breed of Internet connected devices, giving birth to a new wave of wearable devices, home automation appliances, digital signage, and the connecting of everyday objects from our personal and business lives to the Internet. Wearable activity trackers, home cameras, thermostats, television, and other Internet connected devices are all using APIs to send and receive data, establishing entirely new sources of rich and increasingly valuable data from entirely new digitally enabled objects. Giving birth to an Internet of things that would produce entirely new markets for home, surveillance, environmental, agriculture, and other emerging API-driven device data. Devices were telling us the temperature in our homes or the wind speeds on the farm, helping us measure, track, and understand our physical world using Internet connected devices that send data back and forth using expanding wireless networks that were covering every corner of our physical world. APIs weren’t just for expanding your affiliate networks to sell more products, or to publish an image to your blog. You could now deploy global infrastructure to support the needs of humans via mobile applications as well as a growing number of automated connected devices that were being used to further blur the line between the online and offline worlds.

Every Online Shift In Last Twenty Years Have APIs Behind It

Every major online shift of the last twenty years have had APIs behind it. E-commerce, social networks, the cloud, mobile phones, and the Internet of things has changed every business sector, and has forever changed how we live our lives. All of these shifts have contributed to making things like ridesharing and food delivery something we now take for granted. APIs have changed how we buy things, how we communicate and connect online and offline. APIs are transforming every existing business sector, while also creating entirely new business sectors, disrupting many aspects of our lives from how we date, to how we vote in an election. APIs allow for the dismantling of what was, and reassembling it into entirely new digital assets that can be bought, sold, and turn people’s everyday behaviors and activities into something that also can be bought and sold online. While the cloud and their APIs exist just beyond the view of the average consumer or citizen, the commercial, social, mobility, and device enablement is front and center for everyone to help quantify the impact APIs are having in their lives. Each of these seismic shifts introduced by APIs are large by themselves, but it really is the collective momentum they all bring to the table together. Mobile and device applications would be much more difficult to deliver without the cloud, and many of these new types of applications become more of the natural flow of our live because of the social connectivity woven into them. All working together to shift more of our everyday lives online so that they can be quantified, tracked, monetized, and all contributing to a real or perceived forward motion enabled by the Internet.

APIs Are What Makes a Platform

APIs are the Internet enabled pipes behind each of the major platforms dominating the business and political landscape today. Amazon, Facebook, Twitter, Uber, Netflix, Instagram, WhatsApp, Pinterest, Salesforce, Expedia, and other top brands are using APIs to power their businesses and their growth. The public APIs are just the tip of the iceberg when it comes to APIs on their platforms, and for every public API from these companies publish, there are probably 100 others internally and providing partner access to valuable digital assets. While the growth of these platforms over the last twenty years have been driven by their public APIs, internal APIs are also a critical aspect of how their business operates. Enterprise capabilities are increasingly being measured by the quantity and adoption of internal APIs. While these brands have emerged as part of the last wave of Internet enabled businesses over the last twenty years, a growing number of more traditional businesses are in the process of also redefining themselves as platforms. Remaking businesses that have been around for fifty or a hundred years, breaking down operations into small API defined bits, then making them available across all of the enterprise applications where they are needed to conduct businesses on any given day. APIs are what turn a web application into a platform, making an application or suite of applications much bigger than the sums of its parts by ensuring digital capabilities are available across many different channels and domains.

Delivery Digital Resources to Web, Mobile, and Device

Most businesses are doing APIs, they just were not doing them in any organized way by 2010, but the introduction of mobile increased the number of domains in which companies were having to deliver data, content, media, and other resources to power a growing number of applications. Web applications were also growing much more complex by 2010, but it was the need to support one or more mobile applications that really pushed APIs into the foreground. Businesses needed to get more organized about not just how they delivered data, content, media, and algorithms to the growing number of applications, but they also needed to get more organized about how they were creating, storing, and making sense of the growing amount of data being generated from all of these new endpoints. Then alongside all of this happening companies were getting asked by partners for access to the same digital resources and data exhaust, while also facing pressure from new players who were being much more public with their API-driven performance, further leveling up the urgency when it comes to doing APIs, but also ensuring they are as performant and easy to put to use as they possibly can internally and externally.

Delivering Digital Resources to Partners & 3rd Party Developers

As the value of data, content, media, and algorithms grows on any platform, the demand for those digital resources increases. As popular platforms like Salesforce, Flickr, Twitter, Facebook, Google Maps, and others grew, the demand for access to their valuable resources by partners and 3rd party developers increased dramatically. This wasn’t exclusive to these platforms. Other companies who were finding success with their web, mobile, and device applications were facing the same challenges. As platforms grew and expanded, so did the demand for access to those resources in a standardized, and oftentimes developers demanded it in a self-service way. Companies needed to meet their own needs for delivering web, mobile, and devices applications, but also the needs of partners and developers who were looking to build the next generation of applications. Amidst all of these real world demand on business to deliver and manage data, content, media, and algorithms across multiple channels, and their desire to satisfy the needs of partners and 3rd party developers, companies are having to get more strategic and organized in how they manage their own digital resources, resulting in many benefits, as well as a whole industry of services and tooling that help companies, organizations, institutions, and government agencies manage their API infrastructure.

Organizational Efficiency, Agility, Reuse, Traceability, and Provenance

There are many more reasons enterprise organizations do APIs, but over the years we have seen some consistent benefits that organizations realize when it comes to operating their businesses using APIs to engage with partners, customers, and the public. Being able to quickly and efficiently discover and apply digital resources where they are needed bring a certain agility to teams when it comes to being able to deliver new web, mobile, and device applications, and more predictably evolve existing ones to meet the changing needs of a platform. Having an organization's digital capabilities defined as APIs, helps improve efficiency and the agility of an organization, but it also allows for the reuse of digital resources, which allows for better traceability, provenance, and management of dependencies across the applications and integrations that depend on them. APIs provide more visibility into who is using data, content, media, and algorithms, by establishing a real-time feedback loop between the backend systems, and the applications putting resources to work, or generating data, content, and media from platform users. APIs aren’t just about developing web, mobile, and device applications, they are about managing the digital resources applied across applications in a more organized way, while developing more awareness of what is actually going on under the hood of a platform and the organization behind.

Recapping the API Fundamentals

To help illustrate what an API is, let’s look at a simple products SaaS platform, where our digital resources are products that we want to sell via an e-commerce platform. To access our products via the web you would simply go to https://example.com/products/ in your browser and you would see a list of products rendered using HTML. Our products API will use a similar URL pattern ofhttps://api.example.com/products, but instead of returning HTML for rendering in a browser, it returns either XML, JSON, or a CSV. Allowing the product catalog to be used to generate HTML, or be rendered into any other application, or integrated into other systems for other uses. The HTML product listing can be easily used in browsers, the CSV product listing can easily be used in a spreadsheet, and the XML and JSON can easily be applied by developers into a wide variety of use cases. This approach can be applied to any digital resource, not just products. For the purposes of this discussion we are focusing on our resource being a product, but this could just as easily be a task list, videos, images, payments, or any other resources that are being made available online. APIs allow us to consistently define a variety of digital resources in a consistent way that leverages HTTP to make data, content, media, and algorithms available using the same low cost infrastructure that powers the web. Early API providers realized quickly that they were going to need their resources available in more than one place on the web.d After building out large web sites or applications, they created affiliate and reseller networks, or were looking to cultivate, grow, and benefit from growing developer communities who were using their APIs in their applications. As early API providers realized the potential of APIs, and continued their investment in developing them they quickly realized that they were going to have to get a little more organized about how they managed their API infrastructure if they were to maximize value within their developer communities, and across their platforms.

The Emergence of API Management in 2006

As APIs were getting started as part of the commerce, social, cloud, and mobile phases of evolution a new approach to managing APIs emerged to help API providers do better APIs. Companies like Mashery, Apigee, and 3Scale emerged to provide a standardized suite of tooling to help secure, manage, and develop an awareness of how APIs were being put to work. By 2012 API management was a viable segment of the tech sector, and by 2016, after some consolidation, API management had become baked into the cloud, and is something that is available as part of AWS, Azure, Google, and other cloud platforms. There are a number of features provided by API management platforms that help API providers better operate their APIs, including securing, rate limiting, and reporting upon API usage, but more importantly API management is about defining and generating value being generated using APIs. API management when used properly helps API providers develop more awareness about how APIs are being used, but also being able to properly manage the value exchanged between a platform and the applications and integrations that exist on top of a platform. Making API management a pretty significant part of the overall conversation when it comes to what is happening across an organizations applications, as well as engagements with partners and the public.

API Management Requires Everyone to Have an API Key

One of the tenets of API management is that they help API Providers secure all of their APIs, requiring all consumers signup to receive a key or token that they need to pass in with each API call they make. Providing a single token that is present in all API requests and responses which is attributable to each API consumer. Allowing access to all digital resources be tracked, rate limited, reported upon, and used to better understand what consumers are doing with the valuable data, content, media, and algorithms being accessed via APIs, while efficiently tracking all data, content, and media being generated by consumers across web, mobile, and device applications. API keys control access to APIs, but this key also acts as a fingerprint on every action a user makes via a platform, allowing their usage to be understood on an individual consumer level, or wider as part of personas, groups, and larger subsections of consumers who are putting APIs to work. API keys have become a ubiquitous way for API providers to providers to define who has access to resources, and how they are putting them to work across applications, but they have also become commonplace for API consumers, and are an expected part on the API onboarding process, when the APIs and the underlying API management layer offers value to consumers.

Giving End Users a Voice with OAuth and Access Scope

Most APIs only require partners and 3rd party developers to signup for access and receive their API keys before they can access API resources, but once they have keys they can usually obtain access to most resources. However, there is an additional layer of security that some API providers employ when there is sensitive platform user data being accessed via an API. Over the last decade OAuth has emerged as a standard for providing not just a set of tokens for accessing an API, but also the scope of usage that is required when it comes to API access to data, content, and media. Facebook, Twitter, Salesforce, and other common API-driven platforms use OAuth to govern access.giviing their end users a voice when it comes to opening up 3rd party access to their data via a platform. Allowing them to authorize the issuing of keys to a 3rd party developer to be able to access their account data, content, and media. While the APIs, and the scope of access are defined by the platform, the end users has a say in who is able to access their data via the platform, initiating what is considered to be the “OAuth dance” to give access to an external platform or application. While imperfect, OAuth provides the best solution we have for brokering the access to valuable platform data by 3rd party developers, while keeping end users in the equation--helping automated and manage the access of valuable resources using publicly accessible web APIs.

Defining the Business of APIs Using Service Plans and API Composition

Since the beginning, API management has provided the ability to group APIs into different access plans, composing exactly the right mix of API defined digital resources and rate limits for consumers. You see this evolution branded on the approaches of software as a service (SaaS) solutions with their free, pro, team, enterprise, and other mix of self-service or direct engagement pricing plans and tiers. API management allows you to put one or many APIs into a plan, put usage limitations on the plan, and then apply it to one or many users, allowing API providers to compose plans for each type of API consumer persona, often providing free public tiers, but then also crafting additional plans for other levels of customers, partners, or even for internal usage across different groups and teams. In my experience all APIs should only be accessed via a well defined, metered, and reported upon usage plan, requiring all developers whether internal, partner, or public 3rd party to obtain API keys that are associated with a well defined API access plan. No API should go without being measured as part of a plan, and no API consumer should be accessing digital resources across an organization without having a key. There can be limited use free and public access to some resources without signing up, but once you need access to more resources, and higher volumes of each resource, history has shown us that you want to have visibility into what digital resources are being made available and who has access, and are putting these resources to work. Without this visibility into how we are putting API resources to work across web, mobile, and device applications, it becomes increasingly difficult to secure and define the value being delivered.

The Role HTTP Methods LIke GET, POST, PUT, DELETE Play

Each page we load on the web uses what is called an HTTP GET—the technical standard for retrieving an HTML page. GET is the most common HTTP method used today, which is all about consumption of data, content, and media. However, there are numerous other HTTP methods available which allow for not just retrieving, but creating, updating, deleting, and other verbs we may need to use in web and API design. The second most common HTTP verb is POST, which is what we use to create a digital resource. We POST to the web and via APIs all the time. We POST contact forms. We POST Tweets. We POST our location. APIs are really good at using HTTP methods, and employ POST (Create), GET (Read), PUT (Update), and DELETE (Delete) to manage the state of data, content, media, and algorithms using the web. When you require that all API consumers must possess a key to access APIs, this ensures that all consumption can be attributed back to each individual consumer, opening up the ability to report on usage by HTTP method, and see and understanding who your consumers are, and who your creators are. Providing one of many dimensions opened up my using API management, but one that tends to reflect a very meaningful to determining who is creating value and who is just consuming.

API Awareness and Reporting from API Management

Layered on top of the authentication, service composition, and rate limiting, API management solutons provide a suite of reporting for API providers to tune into what is happening across the APIs they are serving up. Reporting upon multiple dimensions of consumption, quantifies what each user is accessing, but also breaks things down by the API resource, HTTP method, errors, time frames, geographic region, and many other dimensions that help API providers develop an awareness of what resources are getting used and how they are applied in applications. This awareness is the biggest benefit of doing API management because it makes the API landscape across an organization more observable, helping providers make sense of the growing sprawl that is enterprise API infrastructure. This awareness isn’t just at the API provider side of the conversation, and most API management solutions allow API providers to share reporting with API consumers as well, helping them understand their own usage across their applications and integrations. Helping provide dashboards for API providers and API consumers to see API usage, visualizing the very digital and abstract API layer that exists behind everything we do online each day. Ensure we are in tune with the API infrastructure we depend on each day, and are able to quantify the value that is being generated each day.

Invoicing for Value, Usage and Consumption via API Management

The final feature of modern API management solutions that help define the value being generated by APIs each day is the ability to invoice each API consumer for their usage each month, charging them varying rates based upon their API plan, and how much they have used. As the types of API resources being made available have expanded over the last twenty years, the approaches to pricing, invoicing, and generating revenue from these API resources have expanded as well. While many API providers charge based upon the number of API calls made, many other providers have pushed the boundaries of API monetization to include per megabyte transferred or stored, by timeframe, or other dimension of each resource being served up. The ability to develop and evolve different API plans, meter, report, and then invoice upon how API consumers access APIs has created entirely new types of business and revenue streams from existing businesses. Think about what Amazon has done with the cloud by making servers, storage, databases, and other infrastructure available as a pay for what you use model. APIs have created entirely new ways to generate revenue from social media and other user data or user generated content, with providers like Twitter giving away low volume data access for free, but then charging high volume data consumers based upon what they consume. Invoicing for API usage isn’t just about charging for API usage, and can be used to signal value exchanged, without actually exchanging money, or even be used to pay those who contribute, while charging those who just consume.

Considering the API Management Fundamentals

API management is software you stand up in front of your APIs to secure access to your API resources, meter, measure, and report upon their usage. You do this with what is called a gateway, proxy, or sometimes a connector. API management has evolved to include other elements over the years, but continues to revlve around a portal where publish your APIs, a registration and login flow for developers to sign up when accessing a service, then providing them with the keys and sometimes code they need to actually consume an API. All API consumption is measured, rate limited, and reporting upon in real-time, as well as according to designated billing and subscription cycles. API management stands in between an API provider's digital resources and an API consumers desktop, web, or mobile application and integration. We can stand up an API management solution in front of our products API we showcased earlier, requiring all developers register and obtain keys before they can use, select an access tier, and in some cases put in a credit card to be billed for what they use. API management secures digital resources from unwanted usage, defining the acceptable limits of value being exchanged, rewarding desired behavior, and blocking or shutting off undesired behavior. Leading API providers have mastered the API management layer to benefit their bottom line, and drive the lion share of the value inward towards their companies. Attracting developers with valuable digital resources, but making sure the platform wins when it comes to generating value and revenue. Striking a balance between making digital resources available to exactly the audience you wish, protecting yourself from unwanted access and usage, while also developing insight and awareness about how consumers are putting resources to work.

The Beginning of an Open Data Movement

As the API economy was getting its footing with commerce, social, cloud, and mobile, another movement was occurring that was focused entirely on making public data available from city, county, state, and federal government agencies as well as non-government organizations, non-profits, and other institutions. Around 2010 you began to see more emphasis on publishing data on the web as compressed downloads, or as XML, JSON, CSV, and other machine readable formats. This movement was born out of the tech industry in the San Francisco Bay Area, but quickly spread to cities around the United States, with similar movements emerging in Europe, Australia, and around the globe. Government agencies were looking to stimulate innovation across the private sector with this data, and the tech sector was looking to build the next killer app on the data being made publicly available. By 2013, governments in both the US and UK were mandating that government agencies publish data, and in 2020, while there has been many ups and downs in the open data movement, many public entities are still publishing data online as download and via APIs.

Considering Public Data Download Versus API Consumption

Even before the popular open data movement was picking up speed by 2010, making data available as a single large, or series of large downloaded compressed files was common. When data was available as smaller datasets, they might also be made available as downloadable machine readable files. Along the way, some data stewards who were aware of the emerging world of APIs were also making data available as simple web APIs, returning XML or JSON responses, adding a more advanced query layer. Downloading data as files proved to be effective for developers who have the resources to download and process them, but the process remained disconnected from the stewards of the data, and rarely possessed any kind of feedback loop that would help improve, evolve, and move data forward to meet the needs of consumers—it was a one way street. Most agencies, organizations, and institutions publishing data were unaware of the wider API movement, and the benefits of establishing a connection with consumers. Data consumers with the most technical resources tended to also be the most vocal about keeping things as they were, pushing back on the need for APIs because they had the resources to download and process the data, and didn’t want any friction when it came to accessing the growing wealth of public data resources that were being made available.

Government Mandating Machine Readable by Default

In 2013, then President of the United States Barack Obama issued a directive that top level cabinet agencies In the United States would have to go machine readable by default when it came to publishing data publicly. Instead of publishing data as spreadsheets and PDFs, they needed to publish it as XML, JSON, or CSV. This would have a significant effect on federal agencies, but also trickling down to states and cities, as well as helping lead when it comes to other countries around the world. Demonstrating that by making data available in machine readable ways you would be able maximize the value and usage of the data in both the public and private sector. Making data available in standardized machine readable formats would extend the reach of research, markets, and the many other efforts underway in the public sector. Leveraging the web to make data available for use in web, mobile, and device applications, helping lead the way when it comes to ensuring the web would continue to be an engine for the economy and society. Using the public sector to drive innovation in the private sector, which would then in turn fuel the growth and impact of government through a healthy economy and a growth in tax revenue. Helping make all levels of government machines readable by default so that data can be easily shared between systems, stakeholders, and constituents across the web, mobile, and device applications they depend upon.

Realizing Open Data and Open APIs Are Open for Business

When people speak of open data they most often refer to open being about access. Meaning it is openly available on the web for anyone to download. While this is the intent of most open data providers, it isn’t the intent of many open data consumers. Many wanted it open so that they could freely use these valuable public resources for use in their commercial applications, allowing them to tap this new data resource like it was a mineral, forest, or oil and gas natural resource that had become available in this growing public sector digital landscape. Open data meant open for business when it came to the digital economy, and the growing number of entrepreneurs who needed the public sector to power their apps, providing the life blood of the tech sector, continuing to build on fifty years of public sector investment in the ever growing private tech sector. This arrangement has always benefited the technology sector, but as the value of data increases, the appetite for public data only grew. Resulting in a vocal minority that is extremely passionate about public data being made available, but rarely motivated to contribute back, pay for data, or have anything getting in their way when it comes to accessing the digital resources they had developed an appetite for. Which has had a chilling effect on the overall growth and velocity of the public data movement, actually working against private sector innovation, but is something that the tech sector isn’t always open to seeing and understanding. There has been a lot of money made with commercial applications of public data, but because there isn’t a feedback loop in place, little value ends up back at the source, again operating as a one way street.

Considering Commercial Usage of Public Data With Google Maps

One of the most significant examples of the value extraction that occurs on public data resources can be found with Google Maps. The ubiquitous mapping solution is built on public data, and continues to harvest public data resources to fuel the commerce solution without giving much back beyond "free" usage of the service. Google Maps depends on federal, state, and municipal data to operate, dominating the conversation around travel, transit, neighborhoods, businesses, and many other dimensions of the public sphere. After over decade of growth, Google Maps is a juggernaut, making it something that government agencies, institutions, and organizations often need, even if they simultaneously are feeding data to the giant Google Maps machine. Real time transit feeds, census, business registry, and other public data is essential to the Google Maps machine continuing to operate and grow to cover the entire world at the global level, but all the way down to the neighborhoods within our communities. The often cash strapped agencies and organizations that provide Google with public data are not given any support from Google, a commercial entity, even if continuing to provide the data requires significant resources. It makes sense for public data to remain freely available, but when you have the resources Google does it doesn’t make sense for things to be such a one way street, with all the value being generated for the public sector flowing into and enriching Google. Public sector data should provide a solid foundation for private sector tech innovation, and help stabilize the data life blood of the leading platforms and applications, but this shouldn’t just be flowing one way. Yes, taxes play a significant role in funding the creation of public data, but in the current state of corporate taxes this isn’t a fair argument, and with public data being a living, evolving, and expanding landscape, there needs to be some value going back into the public sphere for it all to keep growing and expanding.

API Realizations From the United States Census Bureau

As long as the United States Census Bureau has been conducting its surveys of the American population it has also been making the data from it available to other government agencies, institutions, organizations, and commercial enterprise entities. In 2011 when I asked the Census Bruearu why they don’t have an API for the massive amount of data it possesses, they simply responded that they are meant to be neutral stewards of the survey data and they couldn’t place any of their own opinions on the public data, something that applying API design might very well do. However, after launching their API shortly after they realized that there was a whole other class of consumers that existed, ones who had lacked the resources to download and process the massive data survey files, and were not able to get at exactly the slice of the census data pie that they needed. With the new web APIs, spreadsheet connectors, and other API driven census survey resources, the Census Bureau was able to reach an entirely new audience. More importantly they were able to establish a feedback loop with consumers that did not exist before with consumers who simply downloaded the large data files for the expansive surveys. When asked if they knew of the interesting projects like the Google Flu Project and other applications that were built on census data, the US Census Bureau said they had no visibility into how the data was being used—something that changed with the introduction of a feedback loop around the current Census APIs. The US Census Bureau had realized the awareness making data available as web APIs introduces, bringing consumers closer, and planting the seeds for more value exchange to be flowing both ways, from the Census Bureau to consumers of the survey data, and then back to the Census Bureau in the form of discussion—eventually, with more investment it could be something significantly more than just feedback from developers.

Managing Public Data Like We Manage Public Lands

In the United States we enjoy an amazing patchwork of public lands and parks that are owned by all citizens and managed by our Department of Interior. These lands are available to every citizen to use. While some are free to use, most require parking and other fees to take advantage of the trails, roads, parking lots, and other ways we engage with public parks and lands. While public lands are available to every citizen, if you are McDonalds and you want to begin selling burgers at the Grand Canyon you have to obtain permits and even share revenue with the federal government. Why are digital public assets different than physical public assets? Public lands require ongoing maintenance even though they already exist. Public data requires maintenance and further investment to keep fresh, alive, and evolving. While Google may pay more taxes than the average citizen, it doesn’t pay more into the government to fund public data than the value it extracts, comparatively to the value extracted by individual citizens. Current views on public data from the tech sector are exploitative and extractive, and do not respect or give back to the valuable digital commons that are emerging. Similar to the reasons why public lands exist in the first place, we need to protect the value that exists within public data resources. There should always be a commercial aspect of why public data exists in the first place, but like public lands it has to be a managed revenue sharing relationship that allows for a shared value expanson between the public and private sector. Times have changed, and it is time that we begin to look at our public data differently, otherwise like our environment we might find us in an unsustainable state.

Stepping Back to Look At The Public API Management Fundamentals

Shifting gears beyond our example products API, helping illustrate the potential of APIs coupled with API management in the public sector, let’s consider how a 311 API can be offered by a municipality or other civic organization, providing programmatic access to non-emergency incidents that occur across a city and the neighborhoods within. Our digital resource in this case is a 311 service request, providing access to a list of requests that have been made via official municipal 311 channels (ie. Phone or Web). Now that we have a published 311 API providing a listing of service requests, we can also stand up an API management solution in front of it, not because we are looking to make a profit, but because we are looking to secure resources, and maximize the value exchange that is going on between municipal API providers and the different types of API consumers that would be interested in the data. We will still offer a limit on free public API calls by IP address, limiting to 100 per day, then requiring users who want more to signup and get access to a free tier that is rate limited at a much higher rate of 1000 per day. We will also then establish higher tiers of access that require API consumers pay for access to APIs beyond 1000 requests per day. Also providing some additional API endpoints providing access to historical data, multi-city sources, or other valuable dimensions for companies and researchers looking to understand what is happening within communities at scale. There is a wealth of knowledge within the 311 service requests across communities, telling stories of class, race, corruption, and what people are needing to achieve a certain quality of life. This data should be available across every city, and the stewards of this valuable data should be able to serve their constituents with free access to the data, but then also generate much needed revenue from commercial providers who are looking to make a buck off of public data. Modern approaches to delivering APIs and API management solutions provide us with the mechanisms we need to make public data available, but also protect the value that it possesses. Ironically, using the same mechanisms that commercial providers are using to protect their own digital resources, and maximize value generation occurring via their own platform.

In 2010 It Was All Tech Leaders Doing APIs

In 2010, when I first started studying the API sector full time I was studying leading API providers like Salesforce, Amazon, Twitter, and Facebook. Enterprise organizations were still doing the much more rigid version of APIs called service oriented architecture (SOA), where the next generation tech companies were beginning to embrace the web and leveraging HTTP APIs to power their platforms. But then, “API as a product” companies like Twilio, Stripe, and others had emerged to show the potential of generating entirely new types of revenue from APIs. These are the companies in which the world of APIs were cultivated from, providing the first looks at how APIs can be used to make not just data available via simple web APIs, but how you could transform existing digital objects into small bite size digital commodities that can be exchanged via HTTP, at a pricing model that was affordable to a wide audience. These are just a handful of the companies who showed that APIs could be used to connect everyday physical objects to the web, helping us better measure weather, water quality, our health and activity. However it would still be a few years before this new formula for defining digital services and making them available to the market would break out of the tech echo chamber, and be something that people would be talking about beyond Silicon Valley. The early days were all about defining a blueprint that was simple enough that others could follow, refining, distilling, and standardizing how we define our digital assets, then securely and cost effectively make them available to the masses.

By 2016 APIs Were Going Mainstream Across All Business Sectors

By 2016, I wasn’t just studying and talking to the tech sector elite when it came to doing APIs. I was talking to the Capital One, Mutual of Omaha, Ford Motor Company, Center for Medicaid and Medicare (CMS), and other public and private sector enterprise entities. The mainstream had woken up to the need for doing web APIs, and doing them well across a large enterprise organization. I do not spend my days convincing people that they should be doing APIs, everyone is doing them. I spend my days educating people about what APIs are, and what the benefits of doing them well are. In a digital world all types of businesses are being forced to become a technology company, and are required to redefine who they are in a market that operates entirely on the web. APIs are a ubiquitous aspect of doing business today, but unfortunately many still don’t fully see the API pipes used beneath everything we do each day. Many developers who are building web, mobile, and device applications with APIs do not fully see and understand what APIs are. APIs have gone mainstream, but there is still a lot of work to be done when it comes to helping companies, organizations, institutions, and government agencies understand how to do APIs simply and consistently, making them something everyone can access and put to work as part of business operations, or within your own personal realm to take more control over your own professional life.

In 2020 APIs Are Making a Mark on Everyone’s Life

In 2020, everyone is being impacted by APIs. APIs power all those apps on our phones, connecting our homes and cars, and are perpetually automating and connecting the world around us. APIs impact our world at the local level by helping us order food for delivery, all the way up to influencing how we vote in an election. APIs are impacting everyone’s life and defining who we are in this new digital world, bit by bit, but also increasingly defining who we are in the physical world. APIs are much like our financial system, where not everyone needs to understand how the entire banking system works, but everyone should have a handle on where their personal data, content, and media lives, and who has access to, much in the same way we have a handle on where our cash, credit, and debt lives, and who has access to this financial layer of our world. Our personal and professional lives are increasingly defined by APIs, and the more awareness of APIs we have, the more control we will have over creating, managing, moving around, and even deleting our digital bits when we want. With an awareness of this layer of our life we will posses more control over who we are online and offline, without it, we are open to exploitation, abuse, and influencing by the myriad of people and corporations who are wielding power online today—giving over a piece of who we are to an increasingly minority group of tech elite.

APIs Are Behind Every Application We Are Using

Every application on our desktops, laptops, and mobile phones use APIs to communicate with the platform behind them. If you proxy your laptop or mobile phone with a piece of software or hardware that records every bit of traffic coming and going, you will see a steady stream of API calls being made behind the scenes. Hundreds or thousands of API calls are made each day with or without you consciously triggering each request. Once you turn on software like Charles Proxy you will see a steady stream of API calls being made from each application even when you aren’t doing anything. Chatting with their platforms, syncing information, pinging home, and doing a variety of other tasks you had no idea were even occurring. Then once you actually begin to surf a variety of web pages you’ll see multiple API calls occurring for each page you visit, calling content and advertising APIs, populating widgets and other elements on the screen. Web and mobile applications behave the same way, using APIs to publish content, and submit your messages, upload your images and videos, allowing you to do what you do on the web each day. It is no exaggeration to say that APIs are everywhere in 2020. Parking meters, gas pumps, cash registers, signs, surveillance cameras, and other common objects are connected to the Internet using APIs, and while everyone doesn’t have to understand how it all works, there should be at least enough awareness that APIs exist to be able to understand that our data and personal information is being shared.

All Of The Top Brands You Know Are Using APIs

If you look down the list of Fortune 500 companies you will see all of the companies working to redefine themselves using APIs in 2020. Starbucks, Nike, Ford, McDonalds, AT&T, Fedex, and others are actively using APIs to deliver the web and mobile apps they are using to expand the reach of their businesses. Some are even embracing public API models, allowing 3rd party developers and trusted partners to take data, content, and media and build entirely new applications with the valuable resources—syndicating, extending, and franchising their valuable digital resources. Top brands and platforms are using APIs to define their businesses, but they are also using them to define their competition and customers. APIs are how we define, track, measure, and reach our customers in a digital world. Management, automation, and orchestration of advertising and marketing is becoming commonplace within any business, demonstrating another evolving API landscape across enterprise organizations that may not immediately reflect internal IT APIs, external public APIs, but are actively defining brands trying to compete online, and the markets in which they are battling for. APIs are how brands are iterating on proven business models lie marketing, advertising, communication, lobbying, finances, and other bedrock aspects of how we do business, pushing them to reflect a much more fast moving stream of ongoing digital transformations that brands must keep up with in order to stay relevant.

With Many Online Platform We Are The Customers

APIs helped usher in an entirely new way to sell software called software as a service (SaaS), defined by their tiered, pay for what you use pricing plans, which were just a reflection of the API management defined pricing plans we defined earlier. All of this expansion increased the number and types of software services available across the landscape while widening the number of services we subscribe to as part of our regular business operations or personal experience. All SaaS companies are using APIs to deliver their services, and a majority of them offer up partner and 3rd party access to those APIs, with APIs also defining the ability for the many different services you use to work with each other via API integration. In this new SaaS economy we are the customer. We are directly sold a service plan that we put in our credit card, and we get the level of service we paid for, subscribing to it for an agreed upon period of time. We are the customer in this SaaS driven world. APIs are defining the underlying resources being delivered, allowing us to access the CRM, document, image, video, or other service as a subscription through a useful web and mobile application, while ensuring they integrate intelligently with each other using their developer APIs. Shifting forever how software gets built, sold, maintained, and evolved, allowing us to subscribe to the services we desire, and effectively manage our digital resources across many different platforms.

With Some Online Platforms We Are The Product

If a platform is offering their services for free via a web or mobile application and via an API, it is likely that instead of being the customer, end users are actually the product. Platforms are increasingly using web and mobile applications to GET, POST, PUT, and DELETE the digital bits of their end users, while then leveraging APIs to make the user generated content, profile, location, and other data available to partners and 3rd party developers. On these free API-driven platforms, the end users are the product. They are what is being harvested and sold. Keeping platform users on a digital hamster wheel, generating valuable digital resources that can then be organized, enriched, and then sold and used for a variety of applicaiton. There are many ways in which our personal data and online activity is organized, aggregated, and packaged up for sale to the highest bidder online. APIs play a significant role in helping define this landscape ranging from powerful advertising APIs on Facebook, Twitter, and Google, to these 3rd party APIs that allow for surveilling, defining, and targeting platform users across the web, and increasingly our physical worlds. All by oneself our data isn’t worth a whole lot, but in aggregate, and over time, this value becomes very significant to defining emerging markets and the next generation of businesses. Leveraging APIs to produce and access the rich data, content, media, and algorithms that are the future commodities of the digital economy.

GET, POST, PUT, DELETE Are Used To Define Us

API providers use HTTP methods such as GET, POST, PUT, and DELETE to define their digital capabilities. As an API consumer, GET, POST, PUT, and DELETE define who I am and what I am building on top of the digital resources I am consuming from one or many different API providers. As an end user of these platforms and applications, GET, POST, PUT, and DELETE are how my digital self is put to work each day on the digital hamster wheel. Defining what we create and consume as we move through our days. Each API defines how we express ourselves online. The POST of a Tweet on Twitter. The POST of a video on Youtube. The GET our bank account balance. The PUT of our LinkedIn profile when we are looking for a new job. We DELETE that Instagram photo from when we drank too much last night. HTTP methods define an API provider, and which methods an API consumer puts to use will tell a story about what they are looking to do across their applications. These two dimensions of the API conversation have the most control over what is happening, and unfortunately most end users are left to just operate within whatever they are given, letting each platform and application developer define who they are, and what becomes of the data, content, and media that gets generated each day. Depending on the type of API resource being served up, such as a message, image, or video, and the platform making it available, each HTTP method can mean a variety of things. Adding a video on Youtube from a birthday party versus from a black lives matter protest. An image of your baby versus an image of your car as it drives through an intersection after being captured via a traffic camera. HTTP methods via APIs are shaping our behavior each day, and defining who we are, as we work and live in an Internet connected world. To help you see the importance of this conversation, think about those psychological tests Cambridge Analytica employed to harvest the data of millions of Facebook users via their API. That should help illuminate the psychological implications of GET, POST, PUT, and DELETE via APIs at scale each day.

APIs Allow Us To Define Personal Digital Assets

APIs are defining our personal digital assets from our bank accounts to our family photos. Our personal lives are spread across Facebook, Instagram, and TikTok. The digital exhaust from our days are captured via the desktop, web, and mobile applications we operated each day. Creating text, images, video, audio, location information, and other virtual bits that tell our narrative each day, and can be used to potentially predict our futures. More and more of our lives are being stored in the cloud and in the datacenter rather than in our closets, photo albums, and garages. The last twenty years of our lives has increasingly been captured, digitized, and made accessible online by us, or by companies, institutions, and government agencies targeting us. APIs define our digital reality as it is created and transmitted into the cloud for further processing, packaging up, and used in other systems and applications. APIs are being used to capture, define, and guide us throughout the world each day. And this is just our personal world. APIs are increasingly being used to capture, define, and direct our professional lives as well. In both our personal and professional capacities an awareness of APIs, and the ability to put them to work can have a profound impact on how much control we have over our personal and professional worlds. APIs can give us an edge in getting hired for a job, or being able to find success with a project at work. APIs can help us take more control over our health care data and our finances. Knowing that a SaaS solution you use has an API, then realizing you can use it to keep two separate services in sync, will incrementally give you more control over your online reality at home and work.

APIs Allow Us To Define Our Corporate Digital Assets

APIs define enterprise digital assets and capabilities. They provide the menu of what is possible when it comes to a company doing business online, and the maturity of APIs and the overall lifecycle employed to deliver APIs will define how fast a company is able to evolve, pivot, and respond to changing business conditions. If you want to see the important role APIs play as the menu for what is possible with an enterprise organization just look at Facebook or Amazon APIs. An API menu for how you change the world, and dominate the technological landscape. APIs are how you break down the monolithic business processes and software development of the last fifty years and you begin redefining, organizing, automating, and orchestrating with them in new and innovative ways. API management is how companies map out this API landscape that exists across an organization, and optimize how those API resources are made available to internal, partner, and public entities for integrating into other systems and applications. APIs are how businesses move their enterprise organizations forward. APIs are how businesses work with their partners. APIs are how businesses define and engage with their customers. The savviest companies out there today have been using APIs to transform themselves to not just stay relevant, but stay ahead of the pack for the last twenty years. It is how Amazon went from an e-commerce bookseller to dominating so many different business verticals while also powering half of the Internet with their Amazon Web Services. APIs are how Twitter goes from a fun social media platform to the global nervous system for the planet. APIs are how corporations are defining themselves, and redefining global markets today.

APIs Are Defining Our Institutional Digital Assets

Government agencies, universities, non-governmental organizations, and other institutions have all been leveraging the web for a greater part of the last twenty-five years, and those who are further along in their journey have realized that APIs are essential to their mission as well.. Most government agencies have APIs of some kind, and you will see more APIs emerging out of universities, first via their libraries, then from IT, faculty, and even student organizations and projects. All of this movement is just one slice of the digital pie when it comes to institutional adoption of APIs, where just like the rest of us, they also are increasingly dependent on APIs for the software as a service (SaaS) solutions they use each day. Leveraging Microsoft, Google, Salesforce, and other API-driven platforms to operate each day. Institutions in 2020 are both API providers and consumers, but many are only becoming aware of this recently. Most institutions are 5-10 years behind the rest of the private sector when it comes to Internet technology, leaving them ripe for vendor lock-in and extraction of value when it comes to the digital resources they possess. It is hard for institutions to say no to Google or Microsoft. Institutions find it difficult to avoid usage of social media, file sharing, and other popular tech platforms. Which leaves them open to some of the same predatory platform practices that the rest of us face when it comes to mining and harvesting of our data, just the stakes are much higher when you are a municipality, federal agency, or higher education institution. In 2020 all institutions are doing APIs, but like many companies they just aren’t doing them well, or as part of any overarching strategy, leaving institutions behind when it comes to the digital evolution.

APIs Are Used To Define Our Public Digital Assets

The public commons began changing substantially after it moved online. In the early days of the World Wide Web (WWW) things were much more free and open, and the web held such promise when it came to information sharing that benefits all, instead of just a handful of corporate entities. While there is a growing number of public data APIs emerging on a regular basis today, we are not seeing the coverage, maturity, and scale that we will need to have the desired impact on both the economy and the institutions behind them. Public data platforms rarely see the investment they need to be successful, with many commercial entities taking more than they give back in direct investment, as well as indirect tax funding of the public commons via trusted institutions. Public data and content is a vital and an increasingly valuable resource in free markets, but we don’t always see the investment and protection required to keep these resources available in a sustainable way while ensuring the value generation is going to the widest possible audience it can. If high value public data makes its way into the markets, it is just as likely to be captured and locked up by commercial entities, than it is to be successfully published as self-service, tiered access resources ensuring those who can’t afford access can get what they need, and those with more resources can also get what they need, but they also are required contribute back in some way that helps ensure an public API is sustainable for every stakeholder. Public data doesn’t just mean all data should just be publicly available. Public data means that it should be accessible and benefit the entire public. We’ve only scratched the surface when it comes to enriching the public commons with rich data, content, media, and algorithms, and with the right investment and protections in place, similar to the ones we’ve public in place for our physical public assets, we can ensure public digital assets continue to be published, evolve, and benefit both the public as well as the private sector.

“Open” and “Public” Can Mean Many Things on the Web

The words “open” and “public” mean different things to different people, both on and offline. Open and public lands will elicit different responses from individuals you ask from Wyoming to New York, as well as if you are asking a small outdoor tour operator and a big oil and gas company. It is even more diverse in a digital word. Google views US Census data very differently than a small grassroots organization trying to improve their local community or region. Open in many entrepreneurial circles means open for business and free in the monetary sense, where others might see it more about access and ability to use. Open source, open data, and open APIs are seen by the majority of consumers as something you take and use, and to a much smaller group as something you contribute back to. It is easy to see the web as about giving, because all the users of the web are getting the information they seek. When in reality there lies a lot more nuance to how data, content, and media gets created, shared, bought, and sold. It is easy to pretend that open or public data should remain free of charge, but data is rarely ever in a finished state, and it costs money to gather, organize, maintain, evolve, and serve up public data. Yes, taxes pay for this to occur, but like public lands and other physical infrastructure, there is further value to be generated and maintained. Value that shouldn’t just be siphoned off to a few, but be something that benefits everyone. Companies are using APIs and public APIs to redefine themselves in a digital age, employing not just APIs, but sophisticated API management practices to generate as much possible value as they can from their digital resources, while keeping the lion share of the value for themselves. There is no reason that public institutions can’t do the same to manage their own digital resources, but also leverage API management to maximize the value exchange, while also keeping the lion share of the value within the institution, helping them better achieve their mission.

Acknowledging the Role Public Resources Play In The Evolution Of Internet Technology

Everything we do on the web today owes a debt to public resources. The Internet was born out of federal government funding, and built on the backs of public universities. Despite popular liberatarian belief that the web is some free market utopia, it is built primarily on the backs of publicly subsidized infrastructure and programs, from our telecommunications network to grant money making companies like Google possible. The technology sector depends on public resources, and it is important that we acknowledge the role that digital data, content, media, algorithms, and other public resources play in making all of this working. Think about what GPS has done for the tech sector. That is what strong and healthy public data commons can do for the future of our economy. APIs are how corporations are redefining the landscape, and it will be how organizations, institutions, and government agencies redefine the landscape as well. Allowing the public sector to employ some of the same practices and tooling to deliver value around digital resources, empowering the next generation of applications, while also generating enough revenue to sufficiently fund the future we all envision. Public digital resources are a critical base for many commercial applications, and it also provides an essential counterbalance for end users of these applications that doesn’t always exist in free markets--requiring other non-commercial entities to step in and help keep everything balanced.

APIs are defining who we are in 2020. Enterprise organizations are aware of this. Developers are aware of this. End users and consumers of online technology are not so aware that every moment of every day is being defined as a never ending stream of API requests. It is widely recognized that data is seen as the “new oil” by entrepreneurs. An endless resource that can be mined, extracted, and transformed into an endless array of new products. The only problem with this analogy is that tihs new resource exists in each of us and the public institutions that we give rise to as a society, which means there is good money to be made in mindlessly mining and extracting value from our personal lives and the public commons in which we all depend on to live our lives. I am not advocating that we employ APIs for everything in our personal worlds and the public space because I think APIs are a good idea. I am advocating this because we have to become more API aware and literate to help us better navigate the digital world around us. APIs already exist--we need to be more aware of them. I am also advocating that we use APIs and some of the same API management practices use by the private sector to help quantify and protect public digital resources from exploitation by commercial interests. Our economy depends on a steady flow of data and other resources from the public sector, and APIs are how we are going to continue to quantify, defend, and evolve this value generation, ensuring that our collective public resources continue to benefit everyone, and not just the few technically savvy entities dominating the global business landscape in 2020.

Pulling The Openapi For Any Api You Are Managing With Postman So That You Can Apply Across The Api Lifecycle

Mon, 23 Nov 2020 00:00:00 +0000

published: true

title: ‘Pulling the OpenAPI For Any API You Are Managing With Postman So That You Can Apply Across the API Lifecycle’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/screen_shot_2020_11_23_at_12.22.32_pm.png

I am using Postman to do more governance on my OpenAPI definitions as part of their API lifecycle. This is a top request of customers I am talking to, so I want to get better at make these individual API lifecycle capabilities much more modular and reusable as Postman collections. In Postman, the OpenAPI is the truth for each API contract throughout the API lifecycle, but each collection has become how you automate each stop along the API lifecycle. Resulting in me needing the OpenAPI for each API I am automating, and being able to pull the OpenAPI truth using the Postman API within each collection I am defining to mock, document, test, and govern each API.

You can find a single collection for pulling the OpenAPI for an API from the Postman API using it’s name (boy that is a mouthful). The collection is designed to abstract away three separate API calls into a single request. Ideally Postman will provide a similar API endpoint, but until that happens I have this collection to help make things easier. The documentation for the collection should provide you with everything you need to get up and running with the collection, pulling the API into a Postman environment for reuse. It is up to you to decide what you will do with the OpenAPI after that, possibly making multiple changes, and then saving back to Postman using the API.

If you have any questions on the collection for pulling the OpenAPI as part of the API lifecycle, feel free to submit a comment for the collection as part of the OpenAPI workspace. I’ll be centralizing the evolution of this collection, as well as other OpenAPI related collections within this workspace. You can also fork the collection and use in your workspaces, and submit back any enhancements you’d like to see as a pull request. If you have any questions that you don’t want to see in the comments for the collection, feel free to email me at info@apievangelist.com.

Automating The Management Of Postman Collections Using A Postman Collection

Mon, 23 Nov 2020 00:00:00 +0000

published: true

title: ‘Automating the Management of Postman Collections Using a Postman Collection’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/screen_shot_2020_11_23_at_9.35.47_pm.png

I am managing more collections via hundreds of different workspaces lately. As part of my work I am needing to make bulk changes to collections based upon a variety of properties. Sometimes I need to find and replace variables, and other times I need to rewrite or append to the descriptions for each collection. Whil I am sure eventually there will be capabilities to do some of this via the Postman interface, I find creating maintenance collections that use the Postman API to do what I need is the quickest way to get what I want, without having to wait for the Postman road map to catch up to my work.

I have a whole list of automated changes I want to make to Postman collections that I am generating from OpenAPI definitions, and to help me quickly work through this list I wanted to create a base collection that I could use to augment with common and some unique scripts for making changes to any collection, and then save the results back using the Postman API. You can find this collection in my public workspace for managing my collection work, where you can fork it and apply to the changes you need to make to collections. You are also welcome to just wait until I get specialized collections built, or feel free to submit a suggestion for a variation that maybe I haven’t considered.

A High Level Look At Api Specifications

Mon, 23 Nov 2020 00:00:00 +0000

published: true

title: ‘A High Level Look At API Specifications’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/aws-s3-stories-crypto-machine-bletchley-copper-circuit.png

I am having an increasing number of conversations around how the leading API specifications work together, and what the role of each are when it comes to various stops along the API lifecycle. To help drive conversations I wanted to create a single blog post I can link to, while also loading up all of my fresh thoughts about API specs into my old brain. All of these API specifications are continuing to see massive adoption across API providers and consumers, and they are all in forward motion being iterated upon by the specification owners, so it helps to pause once or twice a year and take a look at what is going on, and work to understand how all of these API specifications work together (or don’t).

What Are The Leading API Specifications

There are a handful of API specifications that are relevant to delivering APIs in 2020, helping provide a vocabulary for stakeholders in the process to use when describing what each API does, so that a common definition can be applied through the API lifecycle, consistently delivering API infrasture across many teams. Here are the API specifications I am focusing on as part of my API Specification Toolbox conversations.

OpenAPI - A specification for defining the surface area of HTTP 1.1 web APIs using JSON or YAML.
AsyncAPI - A specification for defining the surface area of HTTP 1.1, HTTP/2, HTTP/3, TCP, MQTT, and AMQP APIs using JSON or YAML.
JSON Schema - A specification for defining the underlying models in use for APIs, adopted by both OpenAPI and AsyncAPI.
Postman Collections - A specification for defining executable collections of HTTP 1.1 web APIs for running in services and tooling
Postman Environments - A specification for defining key / value pairs that get applied across collections at execute time.
RAML - A YAML format for describing the surface area of your HTTP 1.1 APIs, and the the underlying objects.
GraphQL - A query language for APIs and a runtime for fulfilling queries on top of data and content.
SOAP - A messaging protocol specification for exchanging structured information across web services.

SOAP has been around for a number of years, but OpenAPI, AsyncAPI, JSON Schema, RAML, and GraphQL have emerged within the latest decade. OpenAPI, AsyncAPI, and JSON Schema have all evolved in concert with each other, where RAML, GraphQL, and SOAP developed in isolation. I am including all of these because I think they need to be discussed in relationship to each other, and we need to work harder to understand the overlapping layers of how we deliver APIs, as well as the common needs when it comes to the lifecycle usage of these API specifications(ie. design, documentation, testing). The success of OpenAPI ,AsyncAPI, and JSON Schema is largely due to overlapping, sharing, and reuse across the specifications, which sets a tone for how things should work we move forward and iterate upon each of these specifications, or introduce new ones.

Reasons Why Each Specification Sees Adoption

There are a number of reasons why API providers and API service providers adopt each of these specifications, but here are a few of the most common reasons why you will find these API specification in use across the enterprise. Hopefully these help you make a decision between what specification you are using.

OpenAPI

Leading API specification for HTTP1.1 web APIs.
Wide service and tooling support for the specification.
Has adopted JSON schema for modeling of objects.

AsyncAPI

Leading API specification for non HTTP1.1 web APIs.
Helps define event and message driven approaches.
Is a sister specification to the OpenAPI specification.
Has adopted JSON schema for modeling of objects.

JSON Schema

You need to validate objects you use across your APIs.
You need to define a common catalog of all your objects.
You need to generate examples for all of your APIs.

Postman Collections

You need to mock your APIs and provide different sets of examples behind HTTP 1.1 APIs.
You need to documentation your APIs, and need to augment with markdown and examples for HTTP 1.1 APIs.
You need to test the request, response, and surface area of your HTTP 1.1 APIs
You need to execute workflows and business use cases across many different APIs.

Postman Environments

You have multiple API base URLs that need to be applied against the same HTTP 1.1 API collection
You have multiple API authentication identities that need to be applied against the same HTTP 1.1 API collection
You have multiple API variables (key / values) that need to be applied against the same HTTP 1.1 API collection

RAML

You are heavily using the Mulesoft API ecosystem.
You are modeling centered in your API design

GraphQL

You possess very data and content centered APIs
Your schema is very large and expansive
APIs are primarily driving single page app development

SOAP

You posses large amounts of legacy enterprise infrastructure that needs supporting
You work in heavily regulated industries that do not move as fast as other industries

Of course, there are many other reasons why developers are adopting these API specifications, but this covers the primary reasons behind API providers and consumers are choosing to adopt each one. While there are other specifications that are relevant to the conversation, to help keep some immediate conversations focused, these are the API specifications that are dominating the conversation today, and for me are relevant to what I am doing as part of the evolution of the Postman platform.

The majority of folks I am talking to are producing OpenAPI, JSON Schema, and Postman collections as part of their regular API operations. These are the API specifications I am most concerned with fleshing out, as well as mapping to a modern API lifecycle. I’ll be producing more storytelling around each of these specifications as I work to flesh out each of their ecosystems and audiences, and work to move some balls forward within the realm of each specification. The next decade is going to be pretty active when it comes to moving these specifications forward, but more importantly, developing the most meaningful services and tooling that help move APIs forward across the API lifecycle which supports each of these API specifications.

Managing The Scope Of Your Openapi

Wed, 18 Nov 2020 00:00:00 +0000

published: true

title: ‘Managing the Scope of Your OpenAPI’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/america-immigration_dumping-ground-city-clouds-waterfront.jpg

Managing the size of an OpenAPI is a common challenge for API development teams. I have regular conversations with teams about the ways in which you can minimize the overall scope of an API, breaking things down into more manageable chunks, while also reducing redundancy—then encouraging reuse. Like other dimensions of the API space there are differing opinions on whether it is better to have one single OpenAPI for all of your APIs, or whether it makes more sense to break things down into well defined bounded contexts. You can see this debate occurring in Monolith vs Microservices, and MonoRepo vs Multi-Repo discussions across the API space. In this game, there are no right or wrong answers, just different approaches that work well for different organizations, teams, and individuals. My goal is to help folks understand the trade-offs by informing them enough to make their own decisions and set into motion API design practices that keep the API factory floor moving along.

The size of your APIs matter. The size of your OpenAPI for your APIs matters. The scope, complexity, and consistency of your APIs will define how easy they are to maintain and put to work. Organizations that do not use OpenAPI struggle with being able to define the API landscape at all, let alone being able to define the scope of individual APIs or groups of APIs. It is common for API providers who are moving into the OpenAPI realm to slam into the brick wall of API scope right away, realizing their OpenAPI definitions are too big to work with in some services and tooling, and become a maze of complexity when it comes to maintenance and consumption. While there are many ways your API design practices can reduce or at least better define the scope of your APIs and resulting APIs, there are a handful of ways you can slice and dice your APIs up to make them easier to manage, or settle in with accepting that there is one monolith OpenAPI to rule them all!

Paths

The design of your resource-centric HTTP APIs will help or hinder your ability to spread the scope of your APIs across multiple files. Depending on the underlying schema model, and the types of resources being made available, as well as the API design training and guidance given to teams, the number and scope of API paths being served up will vary. If you have a coherent resource and sub-resource strategy it becomes easier to break things down into more manageable groupings. Helping break your APIs and the OpenAPIs that define them into much more meaningful and reusable chunks. API paths will define your OpenAPI journey, and define the complexity and breakdown of your API lifecycle, so make sure you are thinking about the design of API paths across all teams throughout your organization.

Methods

How you use your HTTP methods as part of the API design process will help you downstream when it comes to breaking things down in your OpenAPI. Efficient use of GET, POST, PUT, and DELETE will help your OpenAPIs be more coherent, but it can also dictate the scope of other OpenAPI elements like parameters, headers, and the schema used within your request bodies. Your API design choices impact the overall scope of your API, something you might not realize until you've had to spend the time generating or crafting a complete OpenAPI definition for your API. How you express the methods of your API will help you define the scope of your APIs, the OpenAPIs that define them, as well as all the downstream effects of delivering and consuming an API—make sure you have a plan for this layer.

Parameters

Adding to the dimensions of how your API design will define the scope of your API and OpenAPI, your usage of path and query parameters, as well as headers will shape how big or small your APIs are. Having an overall organization API design strategy will help lay the groundwork for the scope of the parameter layer of your APIs. Using common parameters for as much of this dimension will help you reuse parameters across APIs, and centralize them as part of your OpenAPI. Each unique parameter will add baggage to the surface area of an API, which is something that will have to be defined in an OpenAPI, adding to the overall scope of each API and resulting OpenAPI. Parameter sprawl is an easy trap to fall into, but once you spend the time laying out an overall API design strategy it is something that will become much more manageable for your team.

Components

The components object represents the greatest opportunity for API providers to be moving from Swagger 2.0 to OpenAPI 3.0, and begin to get a handle on the scope of APIs, and your OpenAPI. The OpenAPI components object lets you centralize common schema responses, parameters, examples, request bodies, headers, security schemes, links, and callbacks, and then just reference them across your OpenAPI wherever they are needed with a $ref. This component centralization and reuse will become your biggest ally when it comes to managing the scope of your API. Components provide you with a single reusable toolbox of the components you need to apply across each of your APIs, providing an in-file components library that helps you logically define the scope of your API with well planned lego bricks rather than every part of your OpenAPI being a custom block.

Files

Once you begin getting your API design house in order, and get more organized about how you use the OpenAPI specification to map out your API landscape, including having your APIs well tagged with logical bounded contexts, you will become more efficient with breaking your OpenAPIs up into separate files. Leveraging your tag work you can split up monolith OpenAPI documents into separate OpenAPI files which can then be used independently of each other. Helping you load balance the scope of your OpenAPI across many files, providing a well organized selection of smaller, more precise OpenAPIs rather than relying on everything being in a single file. Of course, there are tradeoffs to having to navigate multiple files, but it depends on whether you like your complexity vertically or horizontally--either way you will have to work through the paths, schema, and other parts and pieces.

External $ref

Once you have things broken into separate files you can begin to use external references to each of the moving parts. One place to begin is to break out the schema for your APIs as separate JSON Schema files which can be referenced using an external $ref where they are applied within an OpenAPI, but then you can also use this individual JSON Schema files for validation and other purposes across your operations. External references allow you to break things up into separate files, potentially setting up a centralized components library, which can just be referenced from wherever they are needed. In addition to the schema sprawl introduced by this approach, not all OpenAPI enabled tooling will support external references, requiring the rehydration of files before imported or put to work within services and tooling, further balancing out the tradeoffs around API scope.

Start Managing OpenAPI Scope Today

I know that many folks will just opt for the easier route of having a single monolith OpenAPI, but that will just push the technical debt and complexity down the road. At some point you will have to manage OpenAPI scope and you are better off starting as early as you can. As stated several times, having an organizational wide API design strategy will inform how you manage the scope of your API, but thinking about scope at the path, method, and parameter levels, while leveraging tags and components to optimize the scope of each of your OpenAPIs can bring a lot of benefits. You can still remain a mono OpenAPI, but actively work to manage scope. However, if you have thousands of APIs being developed, iterated upon, and delivered across your organization I highly recommend beginning investing in separate files and external references, as this is where you will begin to see the biggest bang for your buck when it comes to managing the scope of your OpenAPIs.

The scope and complexity of your OpenAPI will be a mirror of your API scope and complexity, which is a mirror of your organizational scope and complexity. If creating and managing an OpenAPI seems unwieldy it is because the design and complexity of your API has become unwieldy. Most people who begin on their OpenAPI journey do so to deliver API Documentation, but then they quickly realize how OpenAPI is much for than just documenting what is going on for consumers. Managing your OpenAPI becomes about managing your API operations and the lifecycle you employ to deliver, iterate upon, and support your APIs. OpenAPI scope is often an illusion based upon operating for years without being able to see the entire API landscape--once you generate that first OpenAPI and begin to get intimate with all the moving parts, you quickly begin to see a bigger picture where scope is more about the lack of visibility and manage practices in the past, which is something that becomes radically amplified by the maturing of your API design and governance strategy—but that is another story.

Using Api Mocks As Part Of An Apifirst Workflow

Tue, 17 Nov 2020 00:00:00 +0000

published: true

title: ‘Using API Mocks as Part of an API-First Workflow’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/mock_story_1.png

I owe an answer on my thoughts about mocking APIs to my coworker Andy, and I don’t want to incur his wrath, so I figured I’d write it up and share wider as I was getting back to him. I don’t have the original question cause it was in a Zoom chat, but it was something to the effect of using mock APIs to move forward the design of an API versus using them to drive application development conversations as well as push testing forward. I’d say that using a mock as part of an API-first design process reflects where I am pushing mocks to work for me more often. I still use them to demo, prototype applications, and as part of API testing, but using them to actually evolve the contract for an API is really where I feel that a mock server will really shine, and can shift how we are delivering APIs. However, there is also a lot of education and awareness needed before I get all the folks I am having conversations with on board with this concept—to help in these efforts I wanted to flesh out a little about what I mean when it comes to API-first usage of mock APIs.

Mocking From Postman Collections

The first part of this conversation involves the concept that you can generate a mock server from any Postman collection. There is a three step process involved with producing a mock API in Postman using a collection. First you create a collection with the path and parameters you would like to use as the definition of the API you are mocking.

Once you have your path you can select examples up near the send button and add a new example for this request. It will take your path with parameters and map to a new example where you can add a JSON example to represent what should be returned by your mock server for an example.

You should now have a single request, and the example that should be returned when the API request is sent. Now you can select to mock the collection, giving it a name, mapping to an environment, making private, and a handful of other settings that will help us manage the mocked representation of our API.

Once complete you will be given a URL for the mock server, which can be added to the environment or collection and used for any requests. This is the baseline of mocking of an API using Postman, which can be used as part of an API-first process, but there are a couple of other dimensions to this that will help truly strengthen mocking as a critical stop along an API-first lifecycle.

Generating Mock Collections From OpenAPI

For the first part of this I am just generating a mock from a standalone collection. Using Postman you can also generate a collection from an OpenAPI definition, and identify the purpose of that collection is for mocking. If you identify it as existing for the purpose of a mock servers when generating it will link the OpenAPI, as well as generate the mock server for you upon generate.

This gives you the ability to derive one or many collections from an OpenAPI, using the OpenAPI as a central truth in the discussion. Then use those collections to power mock APIs, as well as a variety of other usage. Which helps define the difference between OpenAPI and Postman collections, but also demonstrates how they can be used in tandem, opening up other doors in the API-First conversation.

Validating Mock Collections Against OpenAPI

With my mock API collection(s) derived from the OpenAPI, Postman knows that my collections are linked to the API contract and I can choose to validate them, letting me know if my mock collections, or collection used for any other purpose are in alignment with the central contract.

When there are issues it will show them to me, and when the collections are in alignment it will show a green checkbox letting me know each of them are in alignment, helping keep every stop along the API lifecycle in alignment with the central OpenAPI contract.

When issues are found between my collection(s) and the OpenAPI, I can review these issue one by one, fixing them individual, or opt to select all changes and make them all at once, forcing the changes on the collection, treating the OpenAPI as the truth and each collection as an API lifecycle endpoint that needs regular syncing.

These changes are downstream from your OpenAPI, meaning you can only add or remove changes to the generated collection, and not pull changes from your collections into your OpenAPI. This allows you to keep iterating upon the OpenAPI contract, and then push those changes downstream. Next, I will be exploring using how collections can be used as part of the contract iteration process to help clarify what is possible and what is not currently.

Iterating Upon the OpenAPI Contract

Bringing this home to a point where I can intelligently respond to Andy's question about using a mock API as part of the API-first process. Yes, you can use it as part of the API-first conversation. I can generate a collection(s) from my OpenAPI contract and publish a mock server, along with documentation to share with stakeholders that I am collaborating with in an API-first way. This mocked API plus documentation will help make things more tangible and real for the conversation, but it falls short of where I’d like to be with allowing these stakeholders to actually fork and submit pull requests on the collections I am using to drive the mocked API. Meaning that I can’t allow stakeholders in the API-first development process to fork and submit pull requests to iterate upon the design of the API, they can only do this with stops along the API lifecycle like docs, testing etc. If someone wants to submit a pull request on the OpenAPI contract for an API and suggest changes to the structure and design of the API they’d have to do this via Github and make sure the OpenAPI is synced with Postman, and then Postman validation would alert and potentially sync any changes with downstream collection.

Ok, I think I have this loaded up in my old brain now. My immediate answer to Andy was going to be hell yes, just use mocks to drive the API-first development of an API. But, this comes with some limitations (for the time being) that needed to also be noted. We can use the mock to drive the API-first conversation, and we can be generating mocks and docs for each iteration, but we can’t be using collections, and pull / merge requests against a collection to drive and sync with each iteration on the OpenAPI. My initial response is to push on the product team to prioritize this “upstream” sync capabilities, but my experience tells me to slow my roll and think a little more about how this works. For now, using collections to mock APIs, and even using an OpenAPI as the source of truth for these collections, to inform the API-first process. If changes to the design of an API are made, they need to be made to the OpenAPI contract via the Postman interface, API, or via Github where the OpenAPI is synced. I am thinking I am going to need another post to Zoom out on this process and include the GitHub piece, but for the moment I want to make sure I answered Andy’s question, and can think straight about this as I am going into other API-first conversations with customers.

The Multiple Dimensions Of Api Deployment

Tue, 17 Nov 2020 00:00:00 +0000

published: true

title: ‘The Multiple Dimensions of API Deployment’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/america-immigration_dumping-ground-working-on-railroad-2.jpg

API deployment is the OG stop along the API lifecycle, but is still the most underserved when it comes to API service providers providing solutions, and as part of the full lifecycle API management conversation. I remember back in 2011 folks asking me which of the API management providers would help them deploy their APIs, which at the time none of them would, being a pretty regular concern when it came to doing APIs at all, let alone doing them well. I have seen many “API deployment” solutions come and go, and after a decade I see some movements forward, but overall API deployment is still really difficult in ways that just doesn't make sense.

Data APIs

Making data available via simple APIs is the most common type of APIs you will come across out there, and the most common reason for providing as well as consuming APIs. Deploying data APIs are pretty straightforward to do, and there have been a wide variety of services and tooling made available to help developers deliver these types of APIs. Even with the simplicity of this type of APIs, I haven’t seen too many platform successfully monetize a data API deployment solution, or deliver a dead simple, widely adopted, single push of a button solution to delivering data APIs. It really is one of those areas that should have been 100% solved by now, and would be if it was for the incentive models that exists behind most platforms, but also due to some pretty widespread unhealthy practices when it comes to managing and publishing data, making a lot of data pretty messy to work with when it comes to making available data APIs.

Content APIs

These types of APIs often overlap with the world of data APIs, but I want to highlight them separately because of content delivery network (CDN) considerations, as well as movements in the headless CMS world when it comes to enabling dead simple API deployment. I would say that the headless CMS movement is one of the more significant movement forward in this area, and I am looking hard at solutions like Flotiq and Strapi to help me take the next big steps when it comes to deploying APIs from within Postman. Another significant difference between data and content APIs is who these words are meaningful to, and the business prioritization these individuals possess. Meaning data is going to mean more to developers and data folks, where content is going to be more meaningful to marketing and business folks. These things matter when it comes to having the budget to pay for a solution, which is something that ultimately drives what I am talking about when it comes to the overall state of API deployment.

Algorithms and Business Logic

The third main bucket of APIs that usually throws a monkey wrench into the overall API deployment conversation is when there is actual code, business logic, and algorithms being deployed as part of an API. Meaning it isn’t just about pulling data from a database or content from a storage location, and you actually need to have code working some magic behind the scenes, doing calculations, and bringing that secret sauce to what our APIs are doing under the hood. Once you have business logic and algorithms in play, the complexity of an API goes up significantly. Source control becomes much more of a reality, and automation of the API deployment process can become much less streamlined, requiring more resources to make happen. This type of business logic and algorithm crunching often operates along side pulling data from a database and pulling content from a CDN or other system, further adding the complexity, and often times leading to API providers just sticking with hand-crafted APIs done by competent programmers.

Programming Languages

It is the programming language where the API deployment universe widens in ways that create a significant amount of friction. I can’t remember who said the quote, but it is something about there not being too much difference between each programming language, but there are massive differences between each of the programming language communities. This is where 60% of tech dogma exists. The auto-generation of server and client side code has eroded about 10% of the friction that exists here, but since most of them do not actually get you to a deployed API and you still have to wire things up, there is still so much work to be done by developers. Honestly, the programming language layer of the API deployment needs to be abstracted away and removed from the conversation. It has caused more problems than it has solutions over the last decade. As an API consumer I do not give a shit about what language your API is written in, but as an API provider this is often the number one challenge when it comes to streamlining and automating how APIs are deployed. Something that I am hopeful for the API gateway and serverless layers of all of this to help dispose of much of the legacy baggage that exists, which is mostly because of programming languages.

API Frameworks

One layer up from the programming language portion of this is the framework discussion. There are a wealth of frameworks in almost every programming language that will help you quickly bootstrap the deployment of your APIs. Some are simple, while others are complicated. Some are simple, while others are robust. This is an area where your programming language beliefs will drive your decision making process. Whether a frameworks speaks to your API deployment needs will depend on how you view the world through your programming language lens. API frameworks have been around for a long time and aren’t going anywhere soon. They are the favorite way for developers to deploy their APIs, but is something that keeps the API deployment conversation in the hands of the wizards, and not something that is accessible by muggles. Which is one of the reasons why the overall API deployment conversation hasn’t moved much over the years, because the wizards like to keep things within their control and will keep fighting to keep this knowledge and capability out of the hands of normal business folks, and keeping things complicated and abstract is one way we do this.

Compute

Once you have your API defined, next you need to be able to power it, something that was historically done just like web applications using physical servers in racks, but then it migrated to scalable compute instance in the cloud, and then further being evolved upon using API gateways, containers, and serverless. The compute layer underneath APIs will speak to how performant it is, and ultimately what type of load it can handle. It is something that is becoming much more easy to define, scale, and orchestrate with containers and Kubernetes, but we have just begun to see the scalable automation we need to streamline how we deliver APIs, and while Kubernetes is powerful at scale, it still isn’t something that the majority of wizards can put to use when it comes to API deployment. We have all the parts and pieces to solve this aspect of API deployment, we just have to bring it all down to a level that works for everyone across the spectrum.

Serverless

I’d say that serverless makes my top list of positive movements forward in the API deployment conversation. It is abstracting away some of the friction that exists across programming languages, frameworks, as well as the compute layer of deploying APIs. I think we will keep seeing significant advances in the area of API deployment due to more investment in the serverless layer of the cloud, and serverless will move beyond just being a trend and become a core part of how we deploy our APIs. Serverless APIs isaresomething that all API providers should be exploring, and something that they should begin with learning about via the cloud platform of their choice, but then quickly begin looking at how cloud vendor lock-in can be quickly addressed by going multi-cloud and looking to open source serverrless solutions. It will continue being one of the bright spots in this conversation, and helping us make API much more push button and scalable without much additional effort.

Database

The database is a cornerstone of the API deployment conversation, and create, read, update, and delete (CRUD) APIs are ubiquitous. I have seen more database API deployment solutions come and go in the last ten years without much forward motion at all. The deployment of an API from a database should have been completely solved by now and just be part of the fabric of the cloud. Overlapping with the serverless conversation we are seeing the serverless model being applied to the database layer, but even with this evolution we haven’t seen the fruits of all of this emerge when it comes to API deployment. The database layer of API deployment shouldn’t still be of concern after all this time and is something I’ll hopefully do a deeper dive into to better understand, because I know these challenges aren’t technical, and they are likely due to business or political friction.

Storage

Right after the database, having a heavy object storage strategy, as well as a foundation for the content delivery strategy, continues to define the API deployment conversation. Having a coherent, shared, scalable strategy for handling images, videos, files, and other objects as part of API operations continues to slow the advancement of API deployment automation and orchestration. APIs should just deploy and have database and storage built in by default—this is something that developers shouldn’t have to worry about. Like serverless, the storage needs that goes along with API deployment should be abstracted away and made something that is part of the fabric of the cloud. Storage shouldn’t be an afterthought, and always be a well planned portion of how APIs are delivered across an organization.

Gateway

API gateways have been a fixture of the API space for the last couple of years, but in recent years they have gotten smaller and become more baked into the fabric of the cloud with the consolidation of API management space by 2015. Their continued adoption of OpenAPI as a standard has further evolved the API deployment conversation using a gateway. While you often times still have to wire up backend systems and databases, with most modern gateways you can deploy all the scaffolding for an API with a single OpenAPI. However, like other areas, even with all of these advancements, push button API deployment via gateways is still elusive. I’ve done it with AWS API Gateway and Lambda, RDS, or DynamoDB using a Postman collection, but frictionless API deployment using gateways isn't the silver bullet many of us figured it would be out of the gate. With a little more work though I predict in the next couple of years we will see most of the rough edges with API deployment shaved off and the it become much easier to deploy and scale APIs as needed across any cloud platforms.

Protocols

Another area of fragmentation and expansion that continues to impact the API deployment conversation is at the protocol layer. While HTTP 1.1 APIs are the dominant approach to deploying APIs, HTTP/2 and now HTTP/3 are gaining speed, as well as continued expansion into the TCP, MQTT, and AMQP realms. Like other stops along the API lifecycle this expanding API toolbox is putting pressure on the services and tooling space, expanding their road map, but also making it more complex for their customers to put to work. All of these challenges aren’t going anywhere soon, and the need to deploy multi-speed and multi-protocol API experiences is only going to grow. It is just another reality we have to content with when it comes to realizing our dreams of push button API deployment.

Specifications

The great API specification wars of 2011 to 2016 took its toll on the API conversation. While everything has settled on OpenAPI being the API specification you should be using across the API lifecycle, the growth of AsyncAPI represents the protocol challenges I spoke of a little bit ago, and JSON Schema and Postman collections reflect other API lifecycle challenges and opportunities when it comes to making all this work consistently at scale. You can deploy a new data or content API 100% from an OpenAPI definition today. I would argue you can also do this with more algorithmic and business logic APIs with a handful of OpenAPI extensions, but we still lack a viable, widespread, commercial solution to realize our API deployment dreams. Like some of the other areas discussed, we are real close to some complete API deployment solutions driven by API specifications, but there are still a lot of rough edges to smooth off before we are where we need to be.

Source Control

Depending on the combination of building blocks in play to deploy an API, source control may be needed. While some API gateway solutions made be “codeless”, it is likely, even if it is serverless that you will need to have a source control strategy. GitHub, GitLab, and Bitbucket have define this layer of the software development lifecycle, and they’ll continue to define the API deployment lifecycle, but we will need to establish more streamlined approaches that speak to API deployment specifically, catering to the evolution of how APIs are being deployed, scaled, and made available across an organization. Without source control, API deployment will become just as unwieldy as other aspects of our legacy operations that existed before we developed modern source control approaches.

Versioning

Overlapping with source control, managing the deployment of multiple versions of an API, as well as the specifications and code that powers them continues to clog the API deployment pipes. While there are many proven approaches to versioning an API, many teams still struggle with doing consistently and effectively across their fast changing API infrastructure. It will continue to be something that slows push button API deployment, but when service and tooling providers are able to abstract away the challenges with versioning we will see API deployment rise to new levels across all business sectors, taking away the uncertainty that exists around change today.

Pipelines

CI/CD is default when it comes to delivering software in 2020, and that includes API infrastructure. While there are many enterprise organizations who are still playing catch up in this area when it comes to their legacy processes, it has also has captured such a significant mindshare when it comes to API deployment, that I find people have difficulty thinking beyond it. CI/CD is essential for delivering repeatable API deployments that providers and consumer can depend upon, but when it comes to API gateway driven deployments and Kubernetes orchestrated implementations, we are going to have to make sure our definitions of what CI/CD is is elastic. There are other ways to ensure that deployments are repeatable, and like other legacy practices we have to make sure existing definitions of CI/CD do not define the future of API deployment. As of 2020 they are the default mode of deploying reliable APIs, but their use as part of serverless deployments is already seeing shifting views, but this is something that will continue to be blown out of the water when it comes API deployment at scale using API gateways.

Testing

Testing any APIs that are deployed should always be the default. You should never deploy something you don’t have tests written for, and any API deployed into production should always have those test run natively as part of the deployment process. These tests should also be available to developers to execute whenever they need, while also being able to run on a schedule in the cloud from any region around the world. The maturation of the API testing landscape will contribute to us providing push button API deployment solutions in coming years, making all the investment we have been doing over the last five years well worth it. Testing is a critical stop along the API lifecycle, but is also a fundamental ingredient in deploying APIs in a consistent, repeatable, and reliable way.

Integrated Development Environment (IDE)

One important doorway in this discussion involves where the API deployment process is initiated, and for many developers this means being able to make happen from within our IDEs. This is where we live. This is where we spend our days defining each stop along the API lifecycle, so it makes sense that each of the areas of this API deployment should be accessible from within the most common integrated development environments (IDE) out there. While API deployment will increasingly be set into motion via other interfaces by developers and non-developers, APIs will continue to be born from within the IDEs of development teams across many different industries for years to come.s

Command Line Interface (CLI)

All aspects of API deployment should be available via the CLI. Like the IDE, this is where many developers live, but it is also how much of the automation and orchestration of API deployment will be done. CI/CD depends on the CLI, and to reliably automate all the moving parts we’ve discussed, the API deployment process will have to be exposed via the command line, putting each action required right at the fingertips of developers with a single command. Any tools or service that are being employed as part of the API deployment process which does not have a CLI dimension won’t be around for very long, as our need to scale the number of APIs we deploy increases exponentially.

Application Programming Interface (APIs)

Welcome to the inception portion of the API deployment conversation, and recognizing that APIs should be deployable using APIs. Like the CLI, every facet of the API deployment process should be accessible via an API. I can deploy an API from within Postman using the AWS API Gateway, Lambda, RDS, and DynamoDB APIs today. While it would take me a significant amount of work to make this execute reliably with a push of a button using a Postman runner, it is possible. API deployment using APIs exists today, API providers just don’t always see their APIs as just another API resource and treat the same as they do the rest of the software development lifecycle. APIs are essential to us automating and orchestrating the future of APIs—helping us effortlessly deliver the thousands of APIs we are going to need to conduct business in coming years.

API Deployment is Complicated

Not all of these dimensions of API deployment apply in all API deployment scenarios. However, they do reflect the overall complexity involved in moving forward API deployment across many different business sectors which are being impacted by the overall API industry. I can develop entirely new solutions for moving this conversation forward and they won’t mean anything on the ground across enterprise organizations because they aren’t even seen or fathomable in the current state on the ground. Ultimately you have to find the right blend of these building blocks above which makes sense to large enterprise organizations and make incremental adjustments to how they are already doing things. There will be a handful of organizations you will have green field opportunities to move the API deployment conversations, but mostly it will be about understand the existing software development lifecycle and map to hat, slowly evolving and shifting the conversation while investing in the education of developers and architects along the way. It takes time, and understanding the realities on the ground in the moment to actually make change.

I see this list as a sort of bingo card for bringing a new API deployment solution to market. I am finally in a place after a decade of work where I have the opportunity to make a meaningful impact on the API deployment conversation. I don’t think I will have to develop anything. I just have to understand state of things at enterprise organizations, and then find the right mix of API deployment solutions already being developed and delivered to invest in and support. I don’t think Postman needs to build the next great solution. I think Postman has to just stitch together the most meaningful offerings from across our partners and then bake it into the Postman platform in just the right way. Ideally, it is a suite of solutions that cover all of the cloud platforms, the top programming languages, and the leading gateways, helping orchestrate using the most logical mix of services and tooling that is as closely aligned with how organizations are already delivering infrastructure, but then move forward in the most tactical way. It is a pretty exciting time to be in the business, cause I feel like we are experiencing a sort of second or third wind when it comes to not just API management, but the entire API lifecycle, but and more importantly the single most stop along the way—API deployment.

Some Api Specification Toolbox Projects That Will Make An Impact

Thu, 12 Nov 2020 00:00:00 +0000

published: true

title: ‘Some API Specification Toolbox Projects That Will Make an Impact’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/america-immigration_dumping-ground-construction-crane-city.jpg

I have been conducting weekly discussions around API specifications each Friday mornings which I call the API Specification Toolbox. The goal is to just identify ways in which we can drive more discussion and participation in API specifications, focusing on OpenAPI, AsyncAPI, JSON Schema, GraphQL, and Postman Collections. My goal is to help increase awareness of these specifications, but more importantly get more people sharing what they are working on and invest more time and resources into supporting the specifications. I conduct an hour of informal discussions from 8:00 AM to 9:00 AM PST each Friday, and we record as many of our ideas and talking points on Github using issues. Some interesting ideas have emerged from the discussion, and I am taking some of the lowest hanging fruit from these ideas, organizing them, and working with the OAI, AsyncAPI, JSON Schema, and Postman team to try and bring some of these ideas to life. There are many other ideas that didn’t make my starter list of project, but these are the ones I feel are the most important, would have the the greatest impact, and be achievable in small bursts of work, and to help drive some conversation I am having, I wanted to flesh out some of the top ideas here on the blog.

I cleaned up the titles for each of these ideas, and grouped them based upon the type of project, and I would like to flesh out a little more, identify how they overlap and are related—then work to actually push them forward without me doing all of the work. I feel like with some assistance from the wider API community, and some investment from the OAI, AsyncAPI, JSON Schema, and Postman, these projects could make a pretty significant impact on what is going on across the API landscapes. My objective with this post is to show how all of these projects can feed each other, and see what I can convince the community, and each specification to help move forward. Providing me with the words I am going to need to sell these ideas to anyone who will listen, and potentially help make things actually happen.

Mapping The Landscape

I feel this area is one of the cornerstone deficiencies that slows the forward motion of the specifications, services, and tooling. In my opinion, everything is dependent on these four areas, meaning if we can clearly articulate the services, tooling, and people involved with these API specifications, and understand how they are extending the specification, all of these specifications and resulting tooling will lurch forward without any clear vision or coherency. As a community, we need to be able to efficiently profile and publish directories of services, tooling, people, and extensions, and do it in a way that can paint a full picture of what is going on. These are the four core projects I feel are needed to map the API specification landscape.

Service Providers #19 - Identifying, profiling, and publishing a directory of API service providers who have adopted API specifications.
Open Source Tools #7 - Identifying, profiling, and publishing a directory of open source API tooling who developed on top of API specifications.
People #20 - Identifying, profiling, and publishing a directory of people who are working on the specifications, and doing things with them.
Specification Extension #24 - Identifying, profiling, and publishing a directory of specification extensions that are in use across the space.

I pulled together a simple Jekyll single page application that runs 100% on Github for service providers and for tooling, and can apply the same “toolbox” approach to people and extensions. It provides an open format that anyone can fork and replicate, publishing their own full or sub-directory. Next, I wanted to identify a handful of things that will be needed to move these forward to help drive some conversations about more investment in making these directories happen—here is what I am thinking about right now.

Single Page Application - I already have a blueprint for these single page catalogs that are driven using the APIs.json specification, and use Jekyll and Github Pages to allow them to run 100% for free on Github. I have written up the approach, and will be working to apply to these projects as work gets done.
Intake Questionnaire - We need an intake questionnaire for each of these directories that can be used when approaching service providers, and tooling creators, as well as a checklist when profiling people and vendor extensions, making the process that anyone can do to help move things forward.
Discovery - While I have pulled together a seed list for service providers and tooling, there will be a lot more work ahead to find new ones, as well as work to find interesting people and what the different extensions are for each of the specifications, bringing new entries to each of these API specification catalogs.
Outreach - We will need one or many people to actually reach out to service providers and tooling creators, and encourage them to fill out the intake questionnaire, then add the information to the APIs.json data stores behind each of the directories, building out the catalogs and ensuring information is correct.
Research - It will take a significant amount of research to find people who are doing interesting things with API specifications, and discover the extensions that exist out there, with an emphasis and focus on OpenAPI extensions to begin with, demonstrating how folks are extending the specification.
Evangelization - It will take some serious evangelism to help get the word out about these directories as they are being built, pushing the service providers, tooling, and people to help spread the word as they are being engaged, as well as invest in ongoing storytelling and amplification about the directories in an ongoing way.

I am using GitHub to move each of these projects forward. There should be an issue added for each service provider, tool, person, and extension. This provides a feedback loop to be established that people who are working on the catalogs can use to record activity and engage with the community. Each issue can be used to handle intake for each entry, evolve each of them, and provide an ongoing feedback loop for each entry, as well as each catalog. It is going to be tough to put a scope and price tag on all of this work, but it seems like there should be a set amount of hours each month that can be dedicated to this work, and recruiting volunteers as well as a handful of paid workers. As I will demonstrate, this work will lay the foundation for other projects I am advocating for, as well as help strengthen and grow the entire API economy.

Self-Service Virtual Storytelling

The next group of projects I would like to see invested in after mapping out the landscape, involves telling the stories and incentivizing conversations about why API specifications matter, and what people are doing with them. All three of these ideas overlap with each other when it comes to content, topics, and potentially coordination. Making them pretty ripe for doing all together, helping provide a rich set of ongoing conversations about the specifications and what they enable.

Video Interviews #3 - Conducting a regular stream of video interviews with people who are developing the specifications, as well as what people are doing with them across APII service and tooling providers, as well as API providers themselves, and possibly the developers who are integrating with the APIs that use them.
Podcast #25 - This could be a pure audio version of the same video interviews above. Maybe we only do one of the formats, but ideally they are done as video interviews, and the audio gets mixed and published as a podcast, covering a wider spectrum of viewers and listeners who may want to engage in different ways.
Forums & Workshops #8 - Moving beyond just interviews and conversations around the specs and service and tooling space, regular forums could be held that teach people about specifications, tooling, and common practices provided regularly scheduled forums and workshops that help educate the growing API specification community.

These interviews, conversations, and forums could be cultivated from the work above on mapping the API landscape. Bringing in service providers, tooling creators, and people doing interesting things across the space, and getting them involved in doing the interviews, participating in the conversations, and leading the forums and workshops. Here are some of the elements that will be needed to help move some of these things forward as part of these projects.

Participant Discovery - Finding the people who will be engaged in these interviews, conversations, forums, and workshops.
Participant Handling - There will be work to handle the scheduling and coordination needed to keep a steady stream of people.
Topic Discovery - More work is needed to make sure relevant topics are discovered and brought to the table for this storytelling.
Video and Audio Production - There will be regular work to edit and produce each of the videos and podcasts being published.
Publishing and Syndication - Once a video or podcast is ready it will take work to publish and syndicate across relevant channels.
Single Page Application - It would be nice to also publish a single landing page that catalogs all of this content as part of the effort.

I am sure there is a lot more work to be done here, but I am just looking to get the main talking points down right now. Then I will open up for discussion during the API Specification Toolbox office hours, and as part of the conversation I am having with the OAI, AsyncAPI, JSON Schema, and at Postman. I really see this self-service virtual storytelling and the mapping of the API landscape all working together in a virtuous cycle that expands the awareness and literacy around API specifications.

Information and Propaganda

The last grouping of ideas I’d like to see happen. I am dubbing information and propaganda because it is all about further expanding the narrative and visual storytelling that exists around the specification. Rounding of the virtual storytelling above with a regular heartbeat from a newsletter, as well as more static longer form content and visual storytelling that helps reinforce what API specifications deliver, and the impact they are making across the software development lifecycle. Here are the three projects I’d love to see get some investment from the community as well as the API specifications themselves.

Community Newsletter #12 - Putting out a regular newsletter that showcases the specifications, services, tooling, people, and interesting conversations going on across the sector.
10 White Papers #5 - Produce a series of white papers that showcase the impact API specifications make on the API lifecycle on the ground within enterprise organizations, providing an overview of what people need to know.
Propaganda Posters #9 - Add a visual layer to the API specification conversation, developing a series of old school propaganda posters that get people motivated about API specifications, and collecting the different posters defining this time of technology.

I have already started pulling together a handful of pilot newsletters, fleshing out the format and topics—I just haven’t sent them or built the lists. I haven’t done any work on the white papers or propaganda posters, but would like to actually start detailing what they may be about. I think these are great projects to bring together the community and the specifications, while providing a regular drumbeat about the space, and reliable content that newcomers can use to get up to speed, and keepsakes that the most passionate can collect to capture a piece of this moment. To help move these projects forward, here are some of the areas I would need to work on fleshing out a bit more.

Newsletter Format - I have already pulled together a proposed format outline for the newsletter, and pulled together issue one and two, trying to seed this project.
Weekly Newsletter - Providing dedicated resources that would actually pull together the newsletter each week based upon submissions, and get out the door.
White Paper Topics - Pick 10 (more or less) topics for each of the API specification white papers, providing what people are needing to learn about the most.
Find White Paper Authors - Identify the authors who are best suited for actually creating each of the API specification white papers.
Produce White Paper Content - Actually write each of the white papers and then edit them, getting them ready for production.
Publish White Papers - Formerly produce each of the white papers, putting a polish on them and getting ready for public consumption.
Syndication - Invest in actually publishing and syndicating each of the white papers, helping make them available outside the specifications.
Poster Topics - Identify the topics for each of the propaganda posters, picking the most impactful areas in which API specifications are making a difference.
Poster Artist - Find an artist who can actually produce each of the propaganda posters, and provide use with the finished product that will work.
Poster Syndication - Establish a strategy for making the posters available and get them out to the community in a way that will make them memorable and exposed.

These are just talking points I can use to help drive some of the conversations I am having. There will be a lot of things to do that I am not seeing, but I wanted to make a first pass at what the scope of these projects might be. Clearly the newsletter will take an ongoing effort, but the white papers and propaganda will take a heavy one time lift, with light touches after that. While we may not be tackling all of these, I wanted to lay them out in hopes that we can at least tackle one or two of them.

Stabilizing and Amplifying the API Specification Toolbox

The service provider, tooling, people, and extensions directories would help us define the API landscape. This would have a stabilizing effect on the conversation. Helping people understand what exists, but more importantly help us stabilize what will and should come next. Then the virtual events, information, and propaganda will help amplify this increasingly well defined API specification driven landscape. It is crazy that there isn’t an authoritative source of services and tooling, and I am always amazed at how interested folks are into talking and sharing knowledge about the API specifications, with so few outlets for all of this to occur. I am regularly amazed at what I come across going on with OpenAPI, AsyncAPI, JSON Schema, and Postman collections as I do my work as the Chief Evangelist for Postman, and API Evangelist each day. I come across compelling stories coming out of top tier API providers about how they are using all of these specifications, with an appetite for telling these stories, and there are very few specification centric outlets to point them to. There is the API specification conference, which was born out of my event API Strategy & Practice, but beyond this there isn’t anything else dedicated to the growing API specification dimension of the API economy.

OpenAPI, AsyncAPI, JSON Schema, and Postman Collections are defining almost every industry in 2020, mapping out the data, content, algorithms that are powering business today. These API specifications are defining the resources behind the desktop, web, mobile, device, and network applications powering our personal and professional worlds. API specifications have grown significantly over the last ten years, and we are going to have to ramp up our investment in these specs if we are going to take things to the next level and realize the potential of what we have all been calling the “API Economy”. I feel like this list of ideas represents how we can light the fire under all of this and begin stabilizing the conversation and driving it all forward. I just have to figure out how to make it all happen without me carrying the entire load. I am already moving some of these projects forward in one way or another, but to be able to do it at the scale we need I am going to need more people and investment from the specifications, services, and tooling providers. Alright, now back to the dog and pony show around this production, selling it to the people I need to help me move this stuff forward—more to come as I further refine each of these ideas and come up with actual plans for executing on them.

An Api Lifecycle Collection Playbook

Wed, 11 Nov 2020 00:00:00 +0000

published: true

title: ‘An API Lifecycle Collection Playbook’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/1_playbook.png

I have a single API request that is becoming the first call I make on a growing number of Postman ccollections. It is a call to the Postman API to pull the OpenAPI for any API I am managing using Postman. This gives me the OpenAPI contract for an API so that I can move forward as part of a variety of stops along the lifecycle, starting with mocking, documentation, and testing as part of an API-first approach, but then also potentially monitoring, securing, generate client code, or even deployment and management of an API. I am working with a number of API service providers to define Postman collections that go well beyond the common APIs you will find in the Postman API Network, helping develop entirely new categories of collections that are designed to deliver different stops along the API lifecycle, so I am looking to develop a playbook for how you create one of these new types of collections, baking your API services into the Postman platform.

Publish a Team Page

Any partnership with an API service provider and the Postman platform begins with publishing of a team profile to the Postman network. Providing a logo, name, and description for your company, creating your official team profile on the Postman platform—here is my reference API implementation team page from the Postman API network.

You can manage your team profile under your team settings in your Postman dashboard. You can upload a logo, choose a name, and provide a description. When ready, you can choose to make your profile public, choosing a subdomain for your company on the Postman platform.

This is the profile for your API services on the Postman platform, accessible by 13 million developers via their desktop application and on the web. All the collections and template you publish for your services will show up here, and be discoverable via Postman and Google search, providing more exposure for the services you are making available to API providers.

API Lifecycle Collections

Most of the Postman collections available in the API Network are from API providers like Twitter and Salesforce, but as we are ramping up our partnering across the API lifecycle, we are looking to get a lot more DevOps, DevSecOps, and APIOps collections available that push the API network beyond just SaaS and product APIs. As Postman is expanding to become a full lifecycle API management provider it is partnering for many of these stops along the API lifecycle. Postman’s roots as an HTTP client, debugging, and testing tool provides a base, and we’ve rolled out the builder, documentation, mocking, code generation, and other portions of the lifecycle over the years. But, we aren’t about to reinvent the wheel when it comes to API deployment, management, security, and other more established portions of the API lifecycle, and we are depending on you to deliver these capabilities for us. So, let’s walk through the anatomy of an API lifecycle collection that can help you deliver API lifecycle capabilities to our 13 million users.

Anatomy of the API Request

When it comes to an API lifecycle collection, it can have as many API calls to your API service provider API as you desire, but there should be a certain anatomy to a request within an API lifecycle collection that focuses on improving upon the life of the API being designed, developed, or managed, leveraging the OpenAPI definition that exists within Postman, as well as the other building blocks that help bring that API to life. Here are the suggested moving parts of an API you can offer as part of an API lifecycle collection.

Providing a Descriptive Resource - Make your API request as meaningful to the API lifecycle as possible, providing a clear action like monitor, scan, secure, mock, etc. helping the user understand the value being delivered.
Pulls OpenAPI from Postman API - Include details about the name or id of the API in question as part of the API request, so that you can easily pull the OpenAPI contract using the Postman API, providing the bones of the request.
Deliver the API Lifecycle Value - Take that OpenAPI and perform some meaningful stop along the API lifecycle, by mocking, documenting, generating tests, generating SDKs, scanning, fuzzing, securing, monitoring, or any other value to API providers.
Publish Results to Environment - Publish all relevant data I will need from the value generated as part of this API call to a Postman environment for me to use. If you mock or deploy my API, provide me with a base URL, if you test, monitor, or secure, provide me with the results.
Return Appropriate Reponse - Then return an informative response with the appropriate status code, allowing me to easily respond to what just happened when it came to moving my API forward along an established API lifecycle.

You can use this blueprint to create any number of API calls that help move an API forward as part of your organizations well established (or not so well established) API lifecycle. Just ensure that each API request leverages the OpenAPI contract for an API, performs some magic, and then returns all actionable data to an environment so that the results are available at my fingertips within Postman. Allowing me to move forward my API in some meaningful way, then build upon that value using other services you or another API service provider offers, getting me where I want to be with the evolution of my API.

Publish to Network

After you have published your team, and built a useful API lifecycle collection—publish it to the Postman API network. Make sure you provide adequate documentation for your collection, explaining what it does, and how you will need to obtain any credentials, keys, or tokens to make it happen. When ready, publish it to your team profile, making it available to 13 million Postman users.

Repeat this for every API lifecycle capability your company brings to the table. Sure, you can make a single collection for the entire surface area of your service's API, but it is even better if you make single executable representations of each capability, publish to the Postman API network, and enable users to execute using a Run in Postman button from across all of the documentation for your API services.

Explore API Lifecycle Collections in the Network

With all of the API lifecycle capabilities you offer as an API service provider defined as Postman collections and published to the Postman network, anyone using Postman can discover via search and browsing, then add to their own workspaces with a single click. With meaningful titles and robust, informative descriptions for all API lifecycle collections, developers can discover exactly that step they need to move forward their APIs according to the API lifecycle they have defined. Picking from a menu of full lifecycle API solutions from Postman, but also from across a diverse range of API service providers who add value—while leveraging Postman as the factory floor for moving each API from idea to production.

Run in Postman

Once a Postman user finds your API lifecycle collection in the Postman API network, and clicks on the Run in Postman button to add to their workspace, they will need a couple of things before they can actually put to work as part of the API lifecycle. While there may be other details needed, ideally the user only needs two important keys to accomplish whatever they are looking to do as part of managing their APIs.

Adds Postman Key - They will need to obtain their Postman API key from their account and provide, otherwise the Postman API will not be able to be access for pulling the OpenAPI and publishing the environment as mentioned above.
Adds Service Key - Next they will need the appropriate keys for the service being provided. Ideally, like Postman, this is just a single API key, which then provides the second half of this API lifecycle collection coin, connecting the dots.

As an API developer I should be able to onboard with a new collection with as little friction as possible. Ideally, I can add keys for all of the API service providers I use to a single or set of Postman environments, and then apply seamlessly across many different API lifecycle collections. Providing the keys to the API factory floor that turn on all stops along the API lifecycle across all APIs and services being developed, then also leveraging the same environments for storing, aggregating, and acting upon all of the results and exhaust of each stop along the way.

Working With Postman to Deliver the Full API Lifecycle

Postman can’t deliver every stop along the API lifecycle. We are depending on our partners to deliver many of the stops we don’t offer, like deployment, management, and security. We also encourage partners to step up and build upon the services we do, and augment them with industrial grade capabilities that go beyond what we offer. This type of collection might seem like pretty lofty stuff, but it isn’t. Once you begin to realize that our APIs are just managed using APIs, this type of collection makes a lot more sense. Postman collections aren’t just for making requests to a single, or multiple APIs using a runner, monitor, or in the pipeline with Newman. Postman collections are also using APIs to mock, document, test, secure, govern, and deliver every other stop along the API lifecycle. These are the Postman collections I am looking to populate the Postman API network in coming months. Working with partners to help me deliver all of the capabilities our developers are going to need to move their APIs forward in a consistent and scalable way across their teams.

If you have questions about how all this works, feel free to reach out. When it comes to reaching developers with your API services, doing it within the tools they are already using makes a lot of sense. Postman is ubiquitous across startups all thew ay to enterprise organizations. It is how developers are making sense of the rapidly expanding API landscape around them, so it make sense for them to also use to make sense of the API lifecycle. Helping them more rapidly move forward the APIs they are developing, in a consistent way across teams, with quick access to any of the new services they will need to iterate upon the API infrastructure they are delivering. I am going to keep working on this API lifecycle collection playbook until I have a short, concise way to demonstrate how API service providers can reach Postman developers with their API services, using Postman collections that leverage OpenAPI contracts to move each API forward across the factory floor, until it is ready for production.

A Lifecycle For The Api Factory Floor

Tue, 10 Nov 2020 00:00:00 +0000

published: true

title: ‘A Lifecycle for the API Factory Floor’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/america-immigration_dumping-ground-factory-road.jpg

I am pushing forward a handful of conversations with enterprise organizations about how to better formalize their API lifecycle workflow. To help me load up my talking points in my head I wanted to publish a post here on the blog. Using Postman I am finally able to actually execute on my API lifecycle visions in the most tangible way since I started doing all of this in 2010. Postman is a Swiss Army Knife that allows me to approach the API lifecycle in a variety of ways depending on the situation I find myself in. While there are a lot of unknowns when it comes to doing APIs, there are also a lot of common patterns to use, and this is an exploration of these standardized approaches, allowing me to articulate them tomorrow during some discussions.

Workspace

It may seem obvious, but I am finding that having a well defined workspace to define, design, and manage APIs is essential to doing APIs well. Having a single place where you can find the contract and supporting artifacts that define each of the stops along the API lifecycle is needed to do all of this at scale, and consistently collaborate and move APIs forward. There are a handful of elements needed to define an API workspace when you are using Postman.

Name - You'd be surprised at how important the name of an API can be when it comes to making it discoverable and usable by consumers--how you name your API will define a lot about it.
Description - A well written description for an API is your first impression when it comes to onboarding consumers. They shouldn't be a novel, but it should provide enough information for a consumer to understand what it does.
Administrative Team Members - Have your leadership team identified, make sure they are plugged in and know what needs to be done, giving them the control and observability they need.
Collaborator Team Members - Define all the other stakeholders who should have direct access to the workspace, and be able to provide feedback, and iterate upon each of the APIs being delivered.

Do not underestimate the importance of a dedicated workspace that is designed for iterating upon your APIs. You should be able to access your OpenAPI, and the collections driving docs, mocks, tests, as well as all of the orchestration applied using runners and monitors via a single shareable URL. You'd be surprised at how much a workspace can anchor the evolution of your APIs, providing the cornerstone needed to move things forward in a logical, coherent, and collaborative manner.

Machine Readable Contracts

The centerpiece of any workspace should be one or many OpenAPI contracts defining what is possible with an API. Providing the details of each request and response as a single human and machine readable contract, that can be versioned, evolved, and used across the API lifecycle. Postman allows you to place the contract for your APIs at the center of your workspace, providing a dashboard where you can manage and collaborate around the evolution of the two dimensions of your API contracts.

OpenAPI - OpenAPI 3.0 is the default specification in use across the API sector. While its last version, known as Swagger, is still the dominant in usage, OpenAPI 3.0 is being adopted by all of the top API providers--making it the natural choice for defining APIs.
JSON Schema - OpenAPI 3.0 uses the latest draft of JSON Schema, providing the ability to define the underlying objects used as part of API requests and responses, helping stabilize how data, content, and algorithms are made available via the web.

Don't underestimate the power of fully fleshed out OpenAPI definition. An incomplete one will cause all kinds of friction downstream, and a robust, well defined OpenAPI will enrich your documentation, provide robust mock servers, and deliver robust contract, integration, performance, security, and other types of testing. Invest in your OpenAPI definitions, and your JSON Schema, making them discoverable via the one or more workspaces where teams are doing what they do best.

API Contract Management

Even once you publish the first draft of your API contracts, you will then need to iterate upon them, and put them to work driving the different stops along the API lifecycle. Postman provides a variety of features that make the management of APIs more collaborative, and automated, helping reduce the load when it comes to the mundane work of delivering APIs. Here are some of the contract management capabilities available when it comes to moving APIs forward as a team, within one or many known API workspaces.

Naming - How you name your APIs can play a role in how you manage it, helping make it discoverable, groupable, and more manageable.
Filtering - It is easy to filter APIs based upon their name, allowing you to type a couple of characters and have exactly the aPIs you need available for managing.
Version(s) - Managing change is the only way you will be able to get a handle on the rapid iteration of API contracts across teams, providing a framework for moving forward without breaking the past.
Autocomplete - Being able to autocomplete your API contracts helps speed up the evolution of API contracts, putting all of the OpenAPI objects at your fingertips when it comes to moving them forward.
Validation - Knowing in real time whether or not your contract follows the OpenAPI specification to the letter, making sure each of your API contracts will work in other tools and services across the API lifecycle.
Watching - When you are managing many different APIs it helps to be able to follow only the APIs you need to stay in tune with, providing a steady stream of relevant details about what is happening with each contract.
Change Log - You have to be able to stay in tune with what has happened with each API being defined, leveraging the change lot to understand the movement made by each API team without asking each of them for an update.
Comments - The ability to communicate around each API is essential, but being able to comment on the API, as well as the moving parts of the API, will help keep the feedback loop self contained.
Reports - You will need reporting on top of the evolution of your APIs, to help- you stay in tune with how they are being used throughout the design, development, and deployment of your APIs.
Share - API contracts need to be shared. They should be shareable via other workspaces so that they can be worked upon and applied in a variety of different contexts.
Sync - API contracts might also be needed in other services, tooling, and as part of an organizational source control, making GitHub two-way sync a pretty critical link in the chain.

If you have managed an OpenAPI contract for an API you know how much detail work is involved. Having a workspace, with a variety of digital capabilities to help you manage your OpenAPI is essential. It will take you 10X the effort to do it all by hand. Postman helps to put the automation and workflow in the management of OpenAPIs in a way that can be wielded in a variety of different ways depending on how you are looking to deliver your APIs, allowing for teams to do things differently to suit their specific needs.

API Lifecycle Collection

In Postman, from each OpenAPI you can generate one or many collections that help you deliver on a variety of stops along the API lifecycle. The OpenAPI is the truth of each API, but collections are how you realize that APIs potential across the API lifecycle. For each collection, there are a number of ways in which they can be leveraged within Postman to help move forward the conversation around each API, providing a standardized approach to delivering APIs across teams.

Naming - Like your APIs, the naming of your collections will help define the value they deliver and make them discoverable and usable by team members.
Filtering - The name will feed into the filtering of collections as well, allowing you to only see the collections you need, depending on how you have planned the naming of your APIs.
Versioning - You will need to be linking each collection with each version of your APIs, keeping them in lock step when iit comes to managing change across each layer of your API infrastructure.
Validation - With the ability to generate one or many collections, you will need to automate the validation of each collection, ensuring they are always in alignment with the central API contract it serves.
Documentation - Collections in Postman natively have documentation built in, but then you can also use it to publish documentation that can be shared via a URL with any consumer or other stakeholder.
Mock Server - Collections can also be used to generate mock servers, leverage the examples included within each OpenAPI to mock the functionality of each API being delivered so it can be used to test and iterate upon each API.
Contract Testing - The prerequisite and test scripts for each individual collection, folder, or request can be used to actually test the contract for each API, ensuring each API stays in alignment with the contract.
Contract Visualization / Exploration - The Postman visualizer available for each individual API request can be used to make the overall API more visual, explorable, and interactive for consumers trying to put it to work.
Forking / Pull Requests / Merging - The ability for team collaborators to be able to fork, submit pull requests, and merge changes with collections, allows for each aPI to be evolved in a more collaborative way.
Comments - Like the feedback loop around APIs, the moving parts of each collection should allow for commenting, and team engagement, providing a feedback loop for each stop along the API lifecycle.
Share - Collections can be shared between workspaces like APIs, but they can also be shared via URLs and Run in Postman buttons, making each stop along the API lifecycle shareable and reusable.

Each collection reflects the OpenAPI it was generated from, but provides the ability to execute upon a variety of stops along the API lifecycle. The collection provides the details you need to inform the road map of the API, but also publish documentation, power mock servers, and more importantly be able to test all aspects of how an API operates. Each collection represents a portion of the enterprise API factory floor that is in use across teams, driving the delivery of API infrastructure.

Operational Collection(s)

Beyond the lifecycle collection that moves each API forward, there is another operational level collection that helps define an API throughout its lifespan. Providing a blueprint for the overall design and maturity of an API, that can be used to see how well, or how badly an API is progressing--allowing them to move forward in an orderly fashion.

Governance - This collection actually tests the surface area of an API, and ensures the design of the API actually matches what is expected of each API across an organization. Helping test the structure rather than the instance of each API, and just it's request and response.

An API governance collection is just one example of what operational level collections can do. Demonstrating how collections can be used to mature and harden APIs, impacting not just a single API and it's consumers, but also impact overall API operations, and affecting how a platform works, or doesn't work.

API Environments

Environments are used as a governor against each collection used across the API lifecycle. Allowing the base URL, keys, tokens, and other essentials to be abstracted away from each collection, then applied using variables as part of the execution process. Environments help define each stage of an APIs life, using them to design, develop, operate, and manage the maturation of each API.

Mock - Providing a mocked representation of an API as it is being designed, developed, and for use to test an API in production.
Staging - Allowing for an API to be staged and moved from development into production, and verified as it becomes available for use.
Production - Providing the details of an API as it is available in a production environment and available for integration with applications.
Govern - Managing the stage of governance for each API, providing a blueprint of the maturity of each API that is running in production.

Environments allow for separation of each stage of an APIs evolution, as well as the overall maturity of the API. Providing a snapshot of it during design and development, deployment, and while in production, then also keeping track of each of the details along the way helping ensure that APIs are in alignment with the bigger picture across an organization--moving APIs forward in concert, helping keeping the API factory floor operating at full speed.

Observability

THe API lifecycle needs to be observable across all APIs if they are to be moved forward in concert. APIs do not do well when designed, developed, and operated in a silo. Observability is all about monitoring APIs using their existing outputs, capturing the exhaust from their evolution and operation using a handful of mechanisms.

Monitoring - Being able to monitor each aspect of an APIs operation, from contract testing to security testing, everything should be able to be monitored on a schedule from any region.
History - History should be default. You shouldn't have to chase it. The exhaust from the usage of APIs across the lifecycle should be available to use to evaluate the overall health of each aPI.
Activity - The regular activity of the API factory floor should be accessible showing how each API, collections, environments, mock servers, monitors, and other moving parts are being configured.

Platform observability should be default, configurable, and extensible. Allowing for the exhaust and activity from the API factory floor to be used to optimize operations. Helping fine tune the engine of operations by being able to see into what is going on with each API without having to chase things down, ensuring observability isn't an afterthought, and something that is essential to managing the direction you are headed with your API infrastructure at scale.

Automation

Alongside observability, automation should be a default, and part of each stop along the API lifecycle. Automation is how you make the API factory floor move and produce at scale. With Postman, there are three distinct ways in which the API lifecycle can be automated, allowing you to move each API forward, but also keep them aligned and scheduled to produce what is needed to achieve the optimal workflow.

Runners - Each component of the API lifecycle that is defined as a collection should be able to be run manually by a developer as part of each API workspaces.
Monitors - Each component of the API lifecycle that is defined as a collection should be able to be run on a schedule from multiple regions around the world.
Pipelines - Each component of the API lifecycle that is defined as a collection should be able to be run as part of an existing CI/CD pipeline, weaving the API lifecycle into existing operations.

Each API should leverage automation for defining and delivering itself. Moving it forward while also keeping it observable and governed throughout it's life. Automation is the conveyor belts for the factory that keep APIs evolving forward, but also the digital capabilities they enable. Allowing each stop along the API lifecycle to be defined as a collection, then executed manually, on a schedule, or as part of the existing software development lifecycle.

OpenAPI Factory Orchestration

Everything I have covered lays the foundation for the API factory floor, but the teams that operate the factory floor will need to have a certain baseline of knowledge and understanding of the big picture to make it all work. Beginning with having a strong understanding of the OpenAPI specification which specifies the contract for all APIs in motion on the factory floor, but also why quality matters when it comes to positively impacting everything downstream from them across API operations and the applications and integrations that depend on them.

OpenAPI Training - Every team member should have a strong grasp of what OpenAPI is all about and be able to understand why the specification matters when it comes to the API lifecycle.
OpenAPI Quality - Every team member should be able to understand why quality matters when it comes to making it all work, and your API lifecycle is only as good as it's OpenAPI core.
OpenAPI Downstream - Each team member needs to intimately understand what the downstream effects are of a complete OpenAPI, and the friction introduced when you don't deliver a robust OpenAPI for each API.

OpenAPI is essential to the narrative of the factory floor, and the API lifecycle. It is the script you write for the orchestra. If your teams aren't properly trained in the OpenAPI driven details, they will never communicate effectively across your operations. OpenAPI will be the background music for your API operations, and provide the hum of the factory floor dictating how the gears move or do not move when it comes to rolling out new APIs, evolving existing APIs, and ultimately deliver the digital capabilities they enable.

Defining the Lifecycle for Your API Factory Floor

Your OpenAPIs provide the contracts for your businesses API factory floor, and collections provide the unit of execution that bring these contracts to life. Ensuring the factory floor also has the necessary automation and observability it needs to move APIs forward, keeping them producing and delivering as desired. All of this will dictate the life and presence of each API, setting in motion their evolution forward, while keeping them serving the applications and integrations that are designed to support them. All of the knobs and dials articulated above provide the gears you need to build and operate a factory floor at any scale, allowing you to scale up or down as needed to achieve desired results.

Next, I will be taking all of this and applying to specific APIs for customers. I will also be fleshing it out and turning it into relevant training material so I can turn it into a workshop. I need to figure out how I can teach it to teams across an organization in a series of internal workshops that help move an organization forward by making sure teams have the training and tools they need. Now that I have the tools to actually deliver the API factory floor that matches the successful practices I've identified over the years, I am looking to roll it out more formally as part of the conversations I am having. I just need to keep hammering on it and refining it down to something that can be shared with others--helping them see the potential of a well defined API lifecycle delivered across a scalable and consistent factory floor.

Shifting How Api Providers Define What An Application Is When Onboarding And Integrating With Their Apis

Wed, 28 Oct 2020 00:00:00 +0000

published: true

title: ‘Shifting How API Providers Define What An Application Is When Onboarding and Integrating With Their APIs’

image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/america-under-socialism-subway-train-125th.jpg

When you sign up to access most APIs you will have to sign up for an account, create an application, and retrieve a set of keys and / or tokens before you will be able to use the API. This is standard practice that is baked into the home grown as well as Commercial API management solutions available across the landscape. This is a concept that was introduced back in 2005 by Mashery, and is something that hasn’t changed much over the years, despite what we build on top of APIs, and use APIs for having changed significantly over the last decade. In 2020, this relic of API management past needs to evolve if API providers and consumers are going to use APIs to their fullest potential—something API management service providers are going to have to take the lead on, and get back to work innovating and not just being a commodity.

When I sign up for Twitter’s API I have to create an application before I can get the tokens I need to begin using the application. I have to provide a name, purpose, and other details about what that application is—even if I do not fully understand what that is. Most API provider’s notion of an application means web or mobile application, but in reality it can be the “application” of the data, content, and algorithms being made available via an API in a variety of ways from just pulling data to integrating in real-time with other systems. As an API analyst and storyteller I rarely am building an application, but just looking to kick the tires and understand what is going on—being forced to define an application is nothing but a road block for me. I may be a small group of API consumers, but there are many other edge cases which are similar to my situation in that we represent the long tail of API consumption in the unknown unknown API usage quadrant—meaning we are still figuring out why we need an API.

The most significant breakdown in the concept of defining an application is that it is designed for use by developers. In an iPaaS, BPM, and API automated business world, the legacy API management application definition doesn’t work. It is bad enough that an application stands in between a regular user of a platform and getting access to their data, content, and the useful algorithms that exist, but we often also hide all of this in some other completely separate developer area, excluding them from the conversation—instead of just making the API, and access to the API a natural part of the platform account management and settings. This separation between business and integration interfaces reflect decades of division between business and IT groups that needs to be done away with as we move forward in an API-driven business landscape. There is no reason that as a user of an “application”, that I need to venture to another separate section of a platform and create another “application”, just so that I can access the same data, content, and algorithms I am already putting to use via the platform. It just doesn’t pencil out anymore.

This portal and management infrastructure that is serving API consumers who are building web or mobile applications. This usually doesn’t include the other types of applications consumers will be developing like integrations, automation, orchestrations etc., but more importantly it doesn’t serve other complimentary API providers, or API service providers. Meaning, Twilio and Stripe are optimized for their consumers to build applications on each of their respective APIs, but there is very little done to address when SMS and payments are made in a single application across both providers. There is a huge opportunity for API providers to share consumers, and optimize for bundled onboarding, integration, reporting, and other key areas. Additionally, there is very little considerations for how API service providers will be putting APIs to work across many different API providers. Which brings me to why I am writing this. As an API consumer using Postman across many different APIs, the burden is on me to setup applications and obtain keys for all of the APIs I am using, and there are very few API management affordances for me as Postman to help bridge these gaps. Can I go setup a single application and service all of my customers via the Twitter API? Yes, I guess I could argue that Postman is the application, but really developers are just using us as a vehicle to get them to their application or integration. Lots of conversations to be had here, and very few ways these discussions are being facilitated via existing API providers or service providers.

I am in a good place in the industry to move this conversation forward. Both by being at Postman, but also with my relationships to API providers, and specifically the API management service providers. This is a ball I think I will move forward in the next five years, helping push forward this critical construct in the API factory floor which seems to be slowing us down. Sometimes we just have to identify these legacy constraints that exist and encourage a handful of API providers and API service providers to take the lead—then we can help move things forward in a more meaningful way. If we are going to scale all of this API stuff to the levels we envision in our heads, and blow smoke about in our API economy storytelling, then we are going to have to trim a lot more friction off the API onboarding and integration dimensions of our operations. This will help us move from using ten or twenty APIs, to being able to use hundreds or thousands of APIs seamlessly across tens or hundreds of API providers with the same amount of effort. We just have to do the hard work to shift our views of what an application is, moving beyond these legacy web and mobile constructs, realizing that each potential API integration can have a wide range of applications in our digital and physical worlds.

Providing Source Metadata As Part Of The Api Json Index For Data Apis

Tue, 06 Oct 2020 00:00:00 +0000

published: true

title: ‘Providing Source Metadata as Part of the API.json Index for Data APIs’

image: http://kinlane-productions2.s3.amazonaws.com/api_evangelist_site/blog/screen_shot_2020_10_06_at_2.03.22_pm.png

The COVID-19 API resource center we launched back in April at Postman was a success. It generated a lot of traffic and API usage, but also brought in about 75 API submissions from the community. One of the things we learned as part of this work is that more APIs doesn’t always mean better outcomes. More APIs means they have to be properly described and tagged, making them more discoverable, but you also need to make they are all “good” APIs. Which, once you begin to dive in, you realize that it becomes quite a can of worms, but luckily we have the APIs.yaml specification to help us map things out in a way that allows to slowly evolve things from human to machine readable. When it came to the COVID-19 data APIs were were making available one key element of “good” was knowing a little more about provenance behind each of the APIs, helping us see the overlap between all of the APIs, as well as the unique outliers who were providing new and novel resources.

First, to make sure everyone is up to speed, APIs.json / APIs.yaml is a simple machine readable index for defining collections of APIs, and mapping out API operations. In this scenario I am using APIs.json as the core of an API toolbox that runs 100% on GitHub using Jekyll and Github Pages. In addition to indexing the documentation and Postman collections for each API, I am adding a provenance property to help define where the data behind each API originate, producing YAML that looks like this.

Right now the x-sources APIs.yaml property just provides an array of name and urls for each of the data sources behind each API. Mostly just providing a list of URL sources that can be displayed, but it is a property that could be evolved to provide more machine readable capabilities like grouped by domain, rating, public or private sector, and other things you might be considered with when discovering data APIs. Eventually I will add a way for me to easily filter by all of the APIs that pull from Johns Hopkins, but for right now I am just looking to list the sources on the website for each resource center.

Initially I thought this would be a provenance type of property, but really it is just about the source of the data in this moment. I am trying to keep each property focused on solving a specific problem, and answering a meaningful set of questions, and I can always evolve the schema for a property or create new ones for addressing other issues. Right now I am just giving each API a bonus point if they have sources listed—even just one. I am finding that little meta details like this help out in search and discovery, but also are a great way to identify the API owners and curators who actually care about their APIs--which always demonstrates the potential for a better quality API in the end.

Using the APIs.yaml source property to rank COVID-19 and U.S. Election APIs is just the beginning of this renewed work around APIs.json / APIs.yaml. I am working on a number of other human and machine readable properties for a variety of APIs. Helping me map out different layers of the API universe from COVID-19 to banking, and back again. If you have any questions about APIs.json / APIs.yamlhead over to the website to learn more, and if you have any questions head over to the GitHub repo for each of the repositories I have setup to power this API resource centers. Feel free to fork and play with any of the projects, or visit the home repo for my API toolbox prototypes to suggest new types of resource centers, or industry API catalogs