Haitham Raik

Software Architecture Evolution

2022-03-07T12:45:00.002+03:00

Have you ever wondered what is the history of the software architecture? When was the first use of architecture acronym in the software? How microservices benefited from DDD, REST, Containers, etc.? How Architecture documentation and descriptions have been evolved over the time?

To answer all of these questions and more, I have build the below timeline infographic that shows how the architecture evolved over the time and became in its current shape(s):

Software Architecture Evolution Timeline Infographic

Notes:

According to the above its worth mentioning that:

Software Architecture as a concept has been founded in 60s, but the Architecture Style and Architecture Pattern as concepts didn’t exist until the 90s
Architecture Documentation has started by defining views concept in 70s, but it didn’t get importance until the 90s with RUP4+1 and TOGAF
Architecture Documentation had a standard in the 2000s by IEEE1471/ISO42010
TCP/IP and internet was founded in the 70s which was the base for RPC, Client/Server Architectures and eventually all the distributed architecture styles we have today (e.g., Multi-Tiers, Component-based Architecture, SOA, Microservices, etc.)
SQL and ERD founded in the 70s
OO Concepts and OO Programming was already exist even before procedural and functional programming paradigm, but OO didn’t gain popularity until the 80s
Object modeling started in the 90s and UML came as a unification effort for the different modeling techniques founded in that era especially OMT and Booch modeling
Its worth noting that SOLID principles have been evolved over time starting from the 80s and the acronym itself haven’t been coined until the 2000s. The first principle founded was LSP in the 80s, followed by OCP, ISP and DIP in the 90s and the last principle founded was Single Responsibility Principle (SRP) in the 2000s
The 3-Tier Architecture founded as a need emerged from the presence of WWW and HTML to build web applications
SOA is an evolution for the Component-based Architecture that benefited from SOAP (1998), WSDL (2002) and XSD (2001) for more interoperability.
Microservice is an evolution for Domain Driven Design (2003) that benefited from REST (2000), Event-driven programming(1970), containers (1979) and many other techniques for more autonomy, maintainability and flexibility.
Containers was already exist since the 70s but it gained popularity in the Microservices era
REST APIs (founded in 2000) was already exist before SOAP-based webservices (first standard released was WS-I BP in 2004), but REST API didn’t get popularity until 2008. For more information about the APIs history, please check the API Definition and History article

Consolidated Design Rules and Standards for Great REST API

2021-03-04T10:49:00.013+03:00

In a previous thread named Consolidated Design Rules and Standards for Great API, we have explored the consolidated list of rules and standards for designing Web APIs which can be applied for REST and non-REST APIs. This thread builds on the previous thread and deeps dive in REST APIs in particular and lists the most common conventions, rules and standards that makes REST APIs great and will be grouped by the HTTP methods.

At the end of this thread, we will see a step-by-step process to help you in the design for a Great REST API.

What REST really means?

To understand what REST really means, we should first understand what Resource, Representation, Application State, and Resource State mean?

Resource

A resource can be anything that can be stored on a computer; file, a record in a database, a computational algorithm, etc.

Resources need an identification mechanism to give them a unique id that distinguishes them from any other resource. In our case, URL are used to give the resources a distinguished identifications

Representation

As explained, a resource can be a record in a database which can be stored as a set of binary data. But it will be meaningless to transmit database binary records between the consumer and server. It will be more realistic if we can transmit a document that describes the database record (or any other resource) in a machine-readable language. This document is called representation.

A representation is a machine-readable document used to describe a resource for the purpose of communication between the client and server (i.e., consumer and provider). Common representations formats are JSON and XML.

Application State

With the web browsers, the user can open any page and he can surf the web moving from page to another. In this case, the currently open page is considered as the current state of the application (e.g., the application here is the browser itself).

In the world of APIs, if a consumer did a call to get a list of users and then did another call to get the details of one of these users. The application state in this case was the “list of users” and then moved to state “detailed user”.

Application state is maintained on the client and the server doesn’t know anything about a client’s application state

Resource State

If a server maintains a resource called user and its name is Alex and the locked flag is false and somehow it has been updated to Sam and the locked flag to true. In this case, the user resource state has moved from “Alex, not blocked” to “Sam, blocked”.

Resource state is maintained by the service, but the client can get a representation for it from the server.

REST

Representational state transfer (REST), as it names implies, it allows the server to transfer the application state by sending a representation of a resource (e.g., as a response of a GET request) and it also allows the client to transfer the resource state by sending a representation for a resource to the server (e.g., send a PUT request).

REST as an architectural style has been introduced by Roy Fielding in 2000. He defined REST as a set of constraints on top of the word wide web to make the APIs look like the web. The fifth constrain (Uniform Interface) is the fundamental extension to build REST API:

Constraint 1: REST architecture must be built based on Client-Server architecture

Constraint 2: REST architecture must be a Layered system proxies and Firewalls and any other intermediates can be added and removed without affecting the communication between the client and the server

Constraint 3: REST architecture must support Cache

Constraint 4: REST architecture must be Stateless which means that the server should not know anything about the Application State (see the above definition)

Constraint 5: REST API must be built using Uniform Interface. To have uniform interface, the following more constraints need to be applied:

Resources must be identified in the requests using URIs
Resources must be manipulated by sending representations using standard set of HTTP methods (GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH)
Messages (request and response payload) must be self-descriptive by using media-types (for example: application/json and application/xml)
Hypermedia As The Engine Of Application State (HATEOAS) must be supported to allow the client to surf the API similar to end-user surfing the web.HATEOAS means the hypermedias returned by the server works as engines that move the client from application state to another.

Generic Rules

The following rules must be followed regardless what is the HTTP method used:

Generic rules for URL naming

Rule 1: Apply the Security rules defined in previous thread

Rule 2: Apply the Performance and Reliability rules defined in previous thread

Rule 3: Apply the Clarity and Consistency rules defined in previous thread

Rule 4: Apply the Change Management Rules defined in previous thread

Rule 5: URI should identify one and only one resource

Rule 6: Resource can be either a collection or an object

Example for collection: /users
Example for single user object: /users/121

Rule 7: Use plural nouns for resources names. For example: /users

Rule 8: Resource URI MUST NOT end with (/). /users

Rule 9: Refer to a specific object for any resource as a sub-resource

Use following to get user with id 121: GET /users/121

Rule 10: If there is a resource hierarchically related to another resource, it is better to be presented as a sub-resource.

You can use the following to get all the addresses for user 121: Get /users/121/addresses
Or you can use the following to get all addresses for all users: Get /users/addresses

Rule 11: Hyphen (-) can be used to enhance the URI readability

Rule 12: Avoid underscore (_) in URI

Rule 13: Don’t use Uppercase in URI

Rule 14: Avoid Dot (.) in URI

Rule 15: Avoid comma (,) and semicolon (;) in URI

Rule 16: Avoid spaces in URI

Rule 17: Don’t use verbs in URI for CRUD operations

DO NOT USE: GET /getusers

Rule 18: For non-CRUD operations (validate, calculate, check, search, lock, unlock, merge, copy, duplicate, process, etc.), use a verb as a sub-source:

It is ok to use the following to lock specific user /users/121/lock
It is ok to use the following to lock all users /users/lock
It is ok to use the following to search for users: /users/search
It is ok to use the following to merge all the addresses of a specific user: /users/121/addresses/merge

Rule 19: For search operations that may need to fetch data from multiple resources, URI can be defined as a parent resource:

GET /search?q=abc

Rule 20: To define variables in URI, use standard URI Template format as per URI Template standard:

In the following {userId} must be replaced by the consumer with valid user Id value /users/{userId}/addresses

Rule 21: Query parameters can be used for filtration, sorting and pagination:

Filtration example; get all active users: /users?active=true
Pagination example; get page 5 where page size is 10: /users?pageId=5&pageSize=10

Rule 22: Use (&) to separate query parameters

Rule 23: Apply one of the following versioning identification strategies:

URI Components (preferred option); before the resource (e.g., /v2/users) or after the resource (e.g., /users/v2). Endpoint Redirection pattern can be used to redirect consumers to the new version endpoint
Custom HTTP header. For example: API-version: 1
Request parameter; e.g., users?v=1

Rule 24: Avoid introducing new media types for each version because it may reduce the API interoperability. The following examples are not recommended:

application/json;v1
application/vnd.usr+json;v1

Rule 25: Apply the following techniques to minimize the impact on the consumer in case of backward incompatible changes:

Introduce a new version for the new changes
Introduce Service API proxy design pattern for existing customers. API proxy retains the original API and contains logic to accommodate the changes.
In case of endpoint change, apply Endpoint Redirection design pattern using Location header to redirect the consumer to the new endpoint. In this case, the server must return status code (301 Moved Permanently)
Apply Termination Notification design pattern to announce version has been deprecated and announce the date of old version retirement
Build change log to track the versions and changes applied in each version
Once the old service is switched service off, server should return status code (410 “Gone”)

Generic rules for Data Models

Rule 26: You may support more than one data format, but make sure to keep JSON as your primary and default format

Rule 27: If multiple formats are supported, allow consumers to negotiate for a given format using Accept header

Rule 28: Give meaningful entities and fields names and try to use standard vocabulary by consulting the following standards (schema.org, GeoJSON, Microformats wiki, Dublin Core, Activity Vocabulary, FOAF)

Rule 29: Define a schema for the data model (JSON Schema or XML Schema) and Assign the right data type and don’t assign string data type for everything unless it is required. In case of JSON, describe your API using Open API Specs and use $ref to include data models schemas

Rule 30: Fields names must be camel-cased started with a lower case “givenName”

Rule 31: Fields names must start and end with a char

Rule 32: Fields names must use only ASCII alphanumeric characters

Rule 33: make sure to reduce the quantity and restrictiveness of validation logic embedded in the API and keep them generic. Detailed validations in the API can make it coupled to the underlying implementation and accordingly it will be always eligible for change whenever the underlying implementation is changed. See Validation Abstraction design pattern for more details.

Rule 34: Use standards for fields values. Use ISO3166 for countries, ISO4217 for currencies, RFC3339 for dates and times formats, etc.

Rule 35: In case you have created a custom list of media types (although not recommended), your schema must be assigned one of the predefined media types. For example

User JSON schema can be assigned vnd.user+json
User XML schema can be assigned vnd.user+xml

Rule 36: Avoid encoding binary data within the data models using base64. instead , use:

Multipart media types
Separate links to fetch the binary data

Rule 37: Make sure to support hypermedias in the representations for the following purposes

To link entities/resources together For Entity Linking purposes (e.g., related, self, parent, author, subsection, etc.)
To support the navigation for collection pages (e.g., previous, next, first, last)
To support the move between workflow activities (e.g., start, edit, next, previous, etc.)
To support the retrieval for binary data (in case multipart media types are not supported)

Rule 38: Build standard schema(s) using supported data formats (JSON, XML, etc.) for Hypermedias details. Below is a recommended format:

{

“href”: “URI or URI Template for linked entity”,

“rel”: “URI used to identify the relationship type between this entity and linked entity e.g., parent, child, next, previous, etc.”

}

Rule 39: For a list of standard relationship types, refer to Web Linking standard and Link Relations, RFC5005, RFC4685, RFC5829, Microformats Wiki for list of registered relations types and make sure to use lowercase characters for all relation types

Rule 40: In each representation include a self link as follows.

{

“id”: 121,

“userName”: “H1212”,

“firstName”:”Haitham”,

“links”: [

{

“href”: “/users/121”,

“rel”: “self”

}

]

}

Rule 41: Include the link in a consistent manner within your data model. Below “links” is a recommended approach to support entity linking:

{

“userName”: “H1212”,

“firstName”:”Haitham”,

“links”: [

{

“href”: “/users/121”,

“rel”: “self”

{

“href”: “/users/12”,

“rel”: “parent”

{

“href”: “/users/121/addresses/3”,

“rel”: “related”

}

]

}

Rule 42: Avoid removing the links or changing the link relation type because that is not a backward compatible change and API consumers will be affected.

Rule 43: Collections models must provide self link for collection resource itself

Rule 44: Collections entries must also include self links pointing to the individual resources. This will allow the client to get more details about the individual resources

Rule 45: Define standard structure for Collections. For example:

{

“users”: [],

“links”: []

}

Rule 46: Collections should support pagination by providing the total number of pages, link to the next page and link to the previous page and preferable two links for the first and last pages. The following is a response for /users?pageId=5&pageSize=10

{

“users”: [

{

“userName”: “H1212”,

“firstName”:”Haitham”,

“links”: [

{

“href”: “/users/121”,

“rel”: “self”

{

“href”: “/users/12”,

“rel”: “parent”

{

“href”: “/users/121/addresses/3”,

“rel”: “related”

}

]

}

“links”: [

{

“href”: “/users?pageId=5&pageSize=10”,

“rel”: “self”

{

“href”: “/users?pageId=6&pageSize=10”,

“rel”: “next”

{

“href”: “/users?pageId=4&pageSize=10”,

“rel”: “previous”

}

]

}

Rule 47: Build standard schema(s) using supported data formats (JSON, XML, etc) for error reporting and use them across all the APIs for failure conditions only.

One option is to use the standard error details schema proposed in RFC7807
Another option is to build your own custom structure and standardize it across all the APIs (e.g., note that the below structure supports reporting multiple errors):

{ “errors”: [{

“code”: “long-form error code e.g., name_too_long”,

“busMsg”: “Describe the error from business point of view and value should follow the same language requested using Accept-Language header. e.g., Name provided is longer than 16 char”,

“tecMsg”: “Describe the error from technical point of view. E.g., Name length must not be greater than 16”,

“note”: “Note for the consumer developer. E.g., Please refer to following URL for more details: http://apidoc.com”

“recommendation”: “Optional field for a recommendation. E.g., minimize the value length and submit again”

]}

}

Rule 48: return longer-form response codes in the error report payload (see previous rule)

Generic rules for Media-Types

Rule 49: Avoid Application-specific media type (e.g., application/vnd.user+json). Defining custom media types reduces the API interoperability. But if you had to define custom media types, make sure to document them and to map them with the predefined data models as part of the Contract-First design approach (see previous thread for more details). Most importantly, avoid introducing new media types for each version.

Rule 50: You may support more than one data format, but make sure to keep JSON as your primary and default format

Rule 51: Allow consumers to negotiate for a given format using Accept header

Rule 52: Avoid encoding binary data with the data models using base64. instead , use multipart media types

Rule 53: UTF-8 must be the only charset for request and response payload. UTF-8 is already the default encoding for both media types; application/xml and application/json and that should never be changed.

Generic rules for Meta data

Rule 54: Meta-data should be transferred using HTTP headers only

Rule 55: HTTP headers should be used for meta-data only and shouldn’t be used for business-related details

Rule 56: Return Content-Length header must be used in the response

Rule 57: If the request has a payload but it doesn’t include Content-Type header. Return status code (400 Bad Request).

Rule 58: If the request has unsupported media-type defined in Content-Type header, return status code (415 Unsupported Media Type)

Rule 59: If the request has a body but it doesn’t match the media-type defined in Content-Type header, return status code (400 Bad Request)

Rule 60: If the request has a body and it matches the media-type defined in Content-Type header, but the message structure or values are invalid, return status code (400 Bad Request)

Rule 61: In response, return the payload using the same format used in the request payload (e.g., request Content-Type) . If the request has no payload, use the Accept header to allow the consumer to negotiate for his preferred data format (e.g, JSON, XML, etc). If no Accept header is provided in the request, use the default data format (e.g., JSON).

Rule 62: Content-Type header must be used in the response and must match the response body payload (for example: don’t return application/json while the response payload is XML-based)

Rule 63: Accept-Language header must be supported to allow the consumers to request the business details in a specific language if multilingual resources are maintained

Rule 64: if Accept-Language header is not provided return the response using the default language (e.g., en)

Rule 65: Encourage your consumer to use Content-Encoding header in the request to compress the request payload

Rule 66: Support Accept-Encoding header to allow the consumers to get a compressed response payload compression.

Rule 67: Return Content-Encoding header in the response to inform the consumer whether the response payload is compressed or not

Rule 68: Encourage your consumers to send the Date header and the API must always return the Date header in the response in all cases; success or failure.

Rule 69: Support Authorization header for OAuth access token and make sure to validate it with the OAuth provider

Make sure the access token is not expired
Make sure the access token is permitted to perform the requested HTTP method on the requested resource

Rule 70: If consumer didn’t provide or provided invalid credentials, it is recommended to return WWW-Authenticate header to tell the client what kind of authentication is expected

This header must accompany with HTTP Status code (401 Unauthorized)

Rule 71: Return Retry-After header in the response if the consumer exceeded the maximum limit allowed.

This header is usually accompany with HTTP status code (429 Too Many Requests)
It is also recommended to return the following custom headers in as well X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset

Rule 72: If consumer requested unsupported method, return HTTP Status code (405 Method Not Allowed) and return Allow Header to inform the consumer with the supported methods

Rule 73: If User-Agent header is provided, use it for logging and statistical purposes only

Rule 74: Use Link header to provide a URL for a human-readable documentation describing the response payload structure. In case of failure, return, in the Link header, a link to a human-readable document that informs the client how to correct or debug the error. Profile is a standard link relation as per RFC6906

Link: <https://dev.twitter.com/docs>;rel=”profile”

Rule 75: If you want to define a new custom header, do not use X- for “extension” as prefix.

Using X- was an old recommendation and it is not valid any more

Rule 76: Return X-OAuth-Scopes header (inherited from GitHub APIs) to list the scopes for which a token has been authorized

Rule 77: Return X-Accepted-OAuth-Scopes header (inherited from GitHub APIs) to list the scopes that the action requires

Rule 78: Return RateLimit-Limit to inform the consumer about the maximum rate allowed

Rule 79: Return RateLimit-Remaining to inform the consumer about the remaining available requests before reaching the max limit

Rule 80: Return RateLimit-Reset to inform the consumer when the current limit window will be reset

Rule 81: Use Signature header to sign the request and response payload. The value should consist of the signature timestamp (to prevent replay attacks) and the signature value. Here is an example:

Signature: t= 1613907779, v=erwewrwrwerwerwerwrwerwwfsdfsd123wqdad31=

Rule 82: Don’t use HTTP headers to customize the HTTP methods behavior

Generic rules for HTTP methods usage

Rule 83: Avoid using nonstandard custom HTTP methods

Rule 84: Don’t use HEAD Http method. It is useless

Rule 85: Don’t use TRACE and CONNECT, they are only applicable for proxies

Rule 86: LINK and UNLINK methods are not yet standardized and you shouldn't use them

Rule 87: Don’t use WebDAV methods as they are not widely supported.

Rule 88: Use GET for reading purposes or for performing safe non-CRUD actions (e.g., validate, calculate, search, compare, etc.)

Safe actions means no updates on the resources states will happen when calling the service.

Rule 89: Use POST to create new parent objects or to execute unsafe non-CRUD operations (such as lock, unlock, merge, copy, duplicate, process, etc.). Such activities are sometimes called tasks and the resources handling tasks are called controllers or processors.

Rule 90: Use PUT to update an existing object or insert a sub-object for an existing parent object

Rule 91: Use DELETE to delete an existing object

Rule 92: Use PATCH to Modify part of a resource.

Rule 93: Use OPTIONS to retrieve resource’s allowed methods. It is good to support it although it is not popularly used

Rule 94: Use each HTTP method as per its semantic specified in HTTP specs. Don’t try to override HTTP method behavior.

Generic rules for HTTP Status Codes

Rule 95: Don’t use HTTP status codes (301 Found)

Rule 96: Return Http Status code (429 Too Many Requests) if the client exceed the maximum rate limit allowed

In this case it is recommended to return Retry-After header with a value that informs the consumer when he should try again
It is also recommended to return RateLimit-Limit to inform the consumer about the maximum rate allowed
And RateLimit-Remaining to inform the consumer about the remaining available requests before reaching the max limit
And return RateLimit-Reset to inform the consumer when the current limit window will be reset

Rule 97: Return Http Status code (500 Internal Server Error) if client did everything correct but the server could not handle the request because of an internal error. Don’t use 500 if it is a client fault

Rule 98: For 4xx errors, it is recommended to return an error payload in the response body to describe the client mistake

Rule 99: For 5xx errors, it is recommended to return an error payload in the response body to tell the client about the issue and if there is any recommendations (e.g., ask client to retry again later)

GET HTTP Method Rules

Rule 100: Use GET for retrieval only and it should be always safe and idempotent. Resource state after sending the GET request must be exactly the same as the resource state before sending the GET request.

Rule 101: Don’t use get for unsafe operations and in case of network failures, consumers should be able to resubmit the request safely.

Rule 102: Use Get to perform safe non-CRUD actions (e.g., validate, calculate, search, compare, etc.) and use query parameters to pass inputs. For such activities, use a verb as a sub-source. /users/compare

Rule 103: Don’t use GET on the same URI to perform multiple operations. The below examples are not allowed (the below technique is called Tunneling which is not recommended at all)

#Request 1

GET /users

Content-Type: application/json

{op=”validate”, users=[]}

#Request 2

GET /users

Content-Type: application/json

{op=”calc”, users=[]}

Rule 104: If consumer didn’t provide or provided invalid or expired credentials (In Authorization header), return HTTP Status code (401 Unauthorized) and it is recommended to return WWW-Authenticate header to tell the client what kind of authentication is expected

Rule 105: If consumer provided un-authorized credentials, return Http status code (403 Forbidden)

Rule 106: if the requested resource (collection or object) doesn’t exist and the server doesn’t know if the resource was there before or not, return Http Status code (404 Not Found)

Rule 107: if the requested resource doesn’t exist and the server knows that the resource was there before, return Http Status code (405 Gone). This is applicable in case an old version of API has been removed and the server doesn’t know if there is a newer version

Rule 108: if the requested object resource doesn’t exist and the server knows it was there before, return (405 Gone). This is applicable in this case a logical delete mechanism is implemented at the level of the object.

Rule 109: if the requested resource has moved to a different location, return Http Status code (301 Moved Permanently) with Location header value pointing to the new URI. This is applicable in case a client is requesting an old version of the API and the server knows where the new version is (see Rule#25).

Rule 110: If the request has a payload but it doesn’t include Content-Type header. Return status code (400 Bad Request).

Rule 111: If the request has unsupported media-type defined in Content-Type header, return status code (415 Unsupported Media Type)

Rule 112: If the request has a body but it doesn’t match the media-type defined in Content-Type header, return status code (400 Bad Request)

Rule 113: If the request has a body and it matches the media-type defined in Content-Type header, but the message structure or values are invalid, return status code (400 Bad Request)

Rule 114: In response, return the payload using the same format used in the request payload (e.g., request Content-Type) . If the request has no payload, use the Accept header to allow the consumer to negotiate for his preferred data format (e.g, JSON, XML, etc). If no Accept header is provided in the request, use the default data format (e.g., JSON).

Rule 115: if the client sent Accept header with a media type that can’t be supported by the server, return (406 Not Acceptable)

Rule 116: In case of abnormal failures, return HTTP status code 5XX

Rule 117: In case of errors or failure, return error report in the response body describing the issue.

Rule 118: Content-Type header must be used in the response and must match the response body payload (for example: don’t return application/json while the response payload is XML-based)

Rule 119: Query parameters can be used for filtration, sorting and pagination:

Filtration example; get all active users: /users?active=true
Pagination example; get page 5 where page size is 10: /users?pageId=5&pageSize=10

Rule 120: Return status code (200 OK) for a success response of GET request

Rule 121: Don’t use Status code (200 OK) and there is an error in the response body

Rule 122: Avoid range requests using Range Header for implementing pagination (e.g., Range: 1-100)

Rule 123: Return Last-Modified header to inform the consumer when was the last update of the resource

Rule 124: Return ETag header to inform the consumer about the latest version of the resource. ETag can be generated using MD5 hash value

Rule 125: Allow conditional GET request by supporting If-Modified-Since and If-None-Match (if provided by the client):

Client propagate If-Modified-Since with previously retrieved Last-Modified Or Client propagate If-None-Match with previously retrieved ETag
Server validates the value, if condition is true, that means the resource has been updated and server must return fresh representation for the resource with HTTP Status Code (200 OK)
If condition is false, that means the resource is not updated and server must return HTTP Status code (304 Not Modified)

Rule 126: If response integrity is important, make sure to sign the response payload (see Rule#81)

Rule 127: Allow caching for successful GET response using all of the below headers in the response:

For Legacy HTTP1.0 use Date:<start-date> and Expires:<cache-end-date> headers
For HTTP1.1 and above use Cache-Control: max-age=<Seconds>, must-revalidate

Rule 128: For critical data that are not supposed to be cached, disallow caching using all of the below headers in the response:

[For Legacy HTTP1.0 use Pragma: no-cache and Expires:0 headers
For HTTP1.1 and above use Cache-Control: no-cache

OPTIONS HTTP Method Rules

Rule 129: Use OPTIONS to retrieve resource’s allowed methods. It is good to support it although it is not popularly used.

Rule 130: Use OPTIONS for retrieval only and it should be always safe and idempotent. Resource state after sending the OPTIONS request must be exactly the same as the resource state before sending the OPTIONS request.

Rule 131: Return Allow header with a list of supported http methods in the response of OPTIONS request. if we sent the following request, we will get the following HTTP header in the response: OPTIONS /users

Allow: GET, HEAD, PUT, POST, DELETE

Rule 132: When a resource supports the PATCH method, add an Accept-Patch header listing the media types supported for PATCH requests:

#Request

OPTIONS /customers/1

#Response

HTTP/1.1 204 No Content

Allow: POST, GET, PATCH

Accept-Patch: application/json

Rule 133: Add a Link header in the response containing a human-readable document that describes the resource

Rule 134: PATCH request and response should not contain a body

Rule 135: Return HTTP Status Code (204 No Content) in case of success

Rule 136: if the requested resource (collection or object) doesn’t exist and the server doesn’t know if the resource was there before or not, return Http Status code (404 Not Found)

Rule 137: if the requested resource doesn’t exist and the server knows that the resource was there before, return Http Status code (405 Gone). This is applicable in case an old version of API has been removed and the server doesn’t know if there is a newer version

Rule 138: if the requested object resource doesn’t exist and the server knows it was there before, return (405 Gone). This is applicable in this case a logical delete mechanism is implemented at the level of the object.

Rule 139: if the requested resource has moved to a different location, return Http Status code (301 Moved Permanently) with Location header value pointing to the new URI. This is applicable in case a client is requesting an old version of the API and the server knows where the new version is (see Rule#25).

Rule 140: In case of abnormal failures, return HTTP status code 5XX

Rule 141: In case of errors or failure, return error report in the response body describing the issue.

POST HTTP Method Rules

Rule 142: Use POST to create new parent objects. In the below example, a new user will be created:

POST /users

Rule 143: Use POST to execute unsafe non-CRUD operations (such as lock, unlock, merge, copy, duplicate, process, etc.). Such activities are sometimes called tasks and the resources handling tasks are called controllers or processors. For such tasks, use a verb as a sub-source.. For example:

To merge the addresses of user Id 121 use: /users/121/addresses/merge
To lock user with id 121 use: POST /users/121/lock

Rule 144: Don’t use POST on the same URI to perform multiple operations. The below examples are not allowed (the below technique is called Tunneling which is not recommended at all)

#Request 1

POST /users

Content-Type: application/json

{op=”async”, users=[]}

#Request 2

POST /users

Content-Type: application/json

{op=”lock”, users=[]}

Rule 145: If consumer didn’t provide or provided invalid or expired credentials (In Authorization header), return HTTP Status code (401 Unauthorized) and it is recommended to return WWW-Authenticate header to tell the client what kind of authentication is expected

Rule 146: If consumer provided un-authorized credentials, return Http status code (403 Forbidden)

Rule 147: if the requested collection resource doesn’t exist and the server doesn’t know if the resource was there before or not, return Http Status code (404 Not Found)

Rule 148: if the requested collection resource doesn’t exist and the server knows that the resource was there before, return Http Status code (405 Gone). This is applicable in case an old version of API has been removed and the server doesn’t know if there is a newer version or not

Rule 149: if the requested collection resource has moved to a different location, return Http Status code (301 Moved Permanently) with Location header value pointing to the new URI. This is applicable in case a client is requesting an old version of the API and the server knows where the new version is (see Rule#25).

Rule 150: If the request has a payload but it doesn’t include Content-Type header. Return status code (400 Bad Request).

Rule 151: If the request has unsupported media-type defined in Content-Type header, return status code (415 Unsupported Media Type)

Rule 152: If the request has a body but it doesn’t match the media-type defined in Content-Type header, return status code (400 Bad Request)

Rule 153: If the request has a body and it matches the media-type defined in Content-Type header, but the message structure or values are invalid, return status code (400 Bad Request)

Rule 154: if the client sent Accept header with a media type that can’t be supported by the server, return (406 Not Acceptable)

Rule 155: In case of abnormal failures, return HTTP status code 5XX

Rule 156: In case of errors or failure, return error report in the response body describing the issue.

Rule 157: Content-Type header must be used in the response and must match the response body payload (for example: don’t return application/json while the response payload is XML-based)

Rule 158: After creating the resource, return HTTP response code (201 Created) and use one of the following headers:

Use Location header pointing to the newly created object if the response payload is empty
Use Content-Location header pointing to the newly created object if the response payload contains the representation of the newly created resource. This will inform the client that the content can be also retrieved by calling this URI

Rule 159: Avoid returning success code with an error in the body.

Rule 160: Return Last-Modified header for the newly created resource

Rule 161: Return ETag header for the newly created resource. ETag can be generated using MD5 hash value

Rule 162: For bulk creation performed synchronously, return HTTP status code (200 OK) and return in the response body a collection of the newly created URLs.

Rule 163: For bulk posting performed synchronously (for unsafe non-CRUD actions), return HTTP status code (200 OK) and return in the response body a collection of the execution results.

Rule 164: For long processing activities (such as creating a huge bulk of objects), support async processing by using a special purpose resource (lets call it /async-tasks). Upon receiving a POST request, create a new resource of /async-tasks and return HTTP status code (202 Accepted) along with a Location header that includes the URL for the newly created /async-tasks object. For example:

#Request

POST /users/async-tasks

{

“Users”: []

}

#Response

HTTP/1.1 202 Accepted

Content-Type: application/json

Location: /users/async-tasks/1

Rule 165: For Async processing /async-tasks resource must support GET request (refer to GET HTTP method rules for a full list of applicable rules) to allow the consumer to inquire for the current state of the original request and should return one of the following:

For still processing state: Return HTTP status code (200 OK) with a response payload indicating that the request is still under processing
For successfully completed state: Return HTTP status code (303 See Other) and Location header pointing for the creation results.
For a failed processing state, Return HTTP status code (200 OK) with error report indicating what is the reason for the failure.

Rule 166: If request integrity is important, ask your client to send the signature in the header and make sure to validate it (see Rule#81). If the signature is not provided or not valid. return HTTP status code (403 Forbidden) along with an error report in the response body describing the issue.

PUT HTTP Method Rules

Rule 167: Use PUT to update an existing object. Following example updates already exist user with id 121: PUT /users/121

Rule 168: insert a sub-object for an existing parent object. Following example inserts new address under existing user id 121: PUT /users/121/addresses

Rule 169: If consumer didn’t provide or provided invalid or expired credentials (In Authorization header), return HTTP Status code (401 Unauthorized) and it is recommended to return WWW-Authenticate header to tell the client what kind of authentication is expected

Rule 170: If consumer provided un-authorized credentials, return Http status code (403 Forbidden)

Rule 171: if the requested resource (collection or object) doesn’t exist and the server doesn’t know if the resource was there before or not, return Http Status code (404 Not Found)

Rule 172: if the requested resource doesn’t exist and the server knows that the resource was there before, return Http Status code (405 Gone). This is applicable in case an old version of API has been removed and the server doesn’t know if there is a newer version

Rule 173: if the requested object resource doesn’t exist and the server knows it was there before, return (405 Gone). This is applicable in this case a logical delete mechanism is implemented at the level of the object.

Rule 174: if the requested resource has moved to a different location, return Http Status code (301 Moved Permanently) with Location header value pointing to the new URI. This is applicable in case a client is requesting an old version of the API and the server knows where the new version is (see Rule#25).

Rule 175: If the request has a payload but it doesn’t include Content-Type header. Return status code (400 Bad Request).

Rule 176: If the request has unsupported media-type defined in Content-Type header, return status code (415 Unsupported Media Type)

Rule 177: If the request has a body but it doesn’t match the media-type defined in Content-Type header, return status code (400 Bad Request)

Rule 178: If the request has a body and it matches the media-type defined in Content-Type header, but the message structure or values are invalid, return status code (400 Bad Request)

Rule 179: in case of abnormal failures, return HTTP status code 5XX

Rule 180: in case of errors or failure, return error report in the response body describing the issue.

Rule 181: Return status code (204 No Content) for a success response and don’t return a payload in the response body

Rule 182: Avoid returning success code with an error in the body.

Rule 183: Return Last-Modified header to inform the consumer the new last update date time

Rule 184: Return ETag header to inform the consumer about the latest version of the resource. ETag can be generated using MD5 hash value

Rule 185: Allow conditional PUTrequest by supporting If-Unmodified-Since and If-Match headers to prevent concurrent update conflicts:

Client can propagate If-Unmodified-Since with previously retrieved Last-Modified Or can propagate If-Match with previously retrieved ETag
Server validates the value, if condition is true, that means the resource has not been updated and there is no conflict and server must return HTTP Status Code (204 No Content)
If the condition is false, that means the resource is updated and there is a conflict. In this case, the server must return HTTP Status code (412 Precondition Failed) along with the updated values of ETag and Last-Modified headers

Rule 186: If request integrity is important, ask your client to send the signature in the header and make sure to validate it (see Rule#81). If the signature is not provided or not valid. return HTTP status code (403 Forbidden) along with error report in the response body describing the issue

DELETE HTTP Method Rules

Rule 187: Use DELETE to delete an existing object. Following example deletes already exist user with id 121: DELETE /users/121

Rule 188: If consumer didn’t provide or provided invalid or expired credentials (In Authorization header), return HTTP Status code (401 Unauthorized) and it is recommended to return WWW-Authenticate header to tell the client what kind of authentication is expected

Rule 189: If consumer provided un-authorized credentials, return Http status code (403 Forbidden)

Rule 190: if the requested resource (collection or object) doesn’t exist and the server doesn’t know if the resource was there before or not, return Http Status code (404 Not Found)

Rule 191: if the requested resource doesn’t exist and the server knows that the resource was there before, return Http Status code (405 Gone). This is applicable in case an old version of API has been removed and the server doesn’t know if there is a newer version

Rule 192: if the requested object resource doesn’t exist and the server knows it was there before, return (405 Gone). This is applicable in this case a logical delete mechanism is implemented at the level of the object.

Rule 193: if the requested resource has moved to a different location, return Http Status code (301 Moved Permanently) with Location header value pointing to the new URI. This is applicable in case a client is requesting an old version of the API and the server knows where the new version is (see Rule#25).

Rule 194: If the request has a payload but it doesn’t include Content-Type header. Return status code (400 Bad Request).

Rule 195: If the request has unsupported media-type defined in Content-Type header, return status code (415 Unsupported Media Type)

Rule 196: If the request has a body but it doesn’t match the media-type defined in Content-Type header, return status code (400 Bad Request)

Rule 197: If the request has a body and it matches the media-type defined in Content-Type header, but the message structure or values are invalid, return status code (400 Bad Request)

Rule 198: in case of abnormal failures, return HTTP status code 5XX

Rule 199: in case of errors or failure, return error report in the response body describing the issue.

Rule 200: Return status code (204 No Content) for a success response and don’t return a payload in the response body

Rule 201: Avoid returning success code with an error in the body.

Rule 202: Return Last-Modified header to inform the consumer the new last update date time

Rule 203: Allow conditional DELETE request by supporting If-Unmodified-Since and If-Match headers to prevent delete based on invalid state:

Client can propagate If-Unmodified-Since with previously retrieved Last-Modified Or can propagate If-Match with previously retrieved ETag
Server validates the value, if condition is true, that means the resource has not been updated and there is no conflict and server must return HTTP Status Code (204 No Content)
If the condition is false, that means the resource is updated and there is a conflict. In this case, the server must return HTTP Status code (412 Precondition Failed)

Rule 204: If request integrity is important, ask your client to send the signature in the header and make sure to validate it (see Rule#81). If the signature is not provided or not valid. return HTTP status code (403 Forbidden) along with error report in the response body describing the issue

PATCH HTTP Method Rules

Rule 205: Use PATCH to update an existing object partially. Following example updates already exist user with id 121: PATCH /users/121

Rule 206: PATCH is relatively new which may not be yet supported by old tools and network devices.

Rule 207: With PATCH, it is recommended to use standard patch format. For example:

{ "op": "add", "path": "/a/b/c", "value": "foo" }

Rule 208: Advertise support for the PATCH via the allow header of the OPTIONS response.

#Request

OPTIONS /customers/1

#Response

HTTP/1.1 204 No Content

Allow: POST, GET, PATCH

Accept-Patch: application/json

Rule 209: If consumer didn’t provide or provided invalid or expired credentials (In Authorization header), return HTTP Status code (401 Unauthorized) and it is recommended to return WWW-Authenticate header to tell the client what kind of authentication is expected

Rule 210: If consumer provided un-authorized credentials, return Http status code (403 Forbidden)

Rule 211: if the requested resource (collection or object) doesn’t exist and the server doesn’t know if the resource was there before or not, return Http Status code (404 Not Found)

Rule 212: if the requested resource doesn’t exist and the server knows that the resource was there before, return Http Status code (405 Gone). This is applicable in case an old version of API has been removed and the server doesn’t know if there is a newer version

Rule 213: if the requested object resource doesn’t exist and the server knows it was there before, return (405 Gone). This is applicable in this case a logical delete mechanism is implemented at the level of the object.

Rule 214: if the requested resource has moved to a different location, return Http Status code (301 Moved Permanently) with Location header value pointing to the new URI. This is applicable in case a client is requesting an old version of the API and the server knows where the new version is (see Rule#25).

Rule 215: If the request has a payload but it doesn’t include Content-Type header. Return status code (400 Bad Request).

Rule 216: If the request has unsupported media-type defined in Content-Type header, return status code (415 Unsupported Media Type)

Rule 217: If the request has a body but it doesn’t match the media-type defined in Content-Type header, return status code (400 Bad Request)

Rule 218: If the request has a body and it matches the media-type defined in Content-Type header, but the message structure or values are invalid, return status code (400 Bad Request)

Rule 219: in case of abnormal failures, return HTTP status code 5XX

Rule 220: in case of errors or failure, return error report in the response body describing the issue(s).

Rule 221: Return status code (204 No Content) for a success response and don’t return a payload in the response body

Rule 222: Avoid returning success code with an error in the body.

Rule 223: Return Last-Modified header to inform the consumer the new last update date time

Rule 224: Return ETag header to inform the consumer about the latest version of the resource. ETag can be generated using MD5 hash value

Rule 225: Allow conditional PUTrequest by supporting If-Unmodified-Since and If-Match headers to prevent concurrent update conflicts:

Client can propagate If-Unmodified-Since with previously retrieved Last-Modified Or can propagate If-Match with previously retrieved ETag
Server validates the value, if condition is true, that means the resource has not been updated and there is no conflict and server must return HTTP Status Code (204 No Content)
If the condition is false, that means the resource is updated and there is a conflict. In this case, the server must return HTTP Status code (412 Precondition Failed) along with the updated values of ETag and Last-Modified headers

Rule 226: If request integrity is important, ask your client to send the signature in the header and make sure to validate it (see Rule#81). If the signature is not provided or not valid. return HTTP status code (403 Forbidden) along with error report in the response body describing the issue

REST API Design Process

Find the nouns in the user story and model them as Resources
Assign the Resources unique URIs. Use URL naming rules
For Resources that can exist only as part of another Resource, define them as sub-resources. For example: /users/{user-id}/addresses
Build schemas for the resources representations using JSON Schema and/or XML Schema. Use Data Model rules
Define the relationships between the models
Build Hypermedias to link the resources together and include them in the resources schemas. Use Hypermedia rules
Find the verbs in the user story and try to map them to CRUD operation and associate them to the resources
Find the non-CRUD verbs and define them as sub-resources. Use POST method Rules
Build Hyper-medias for workflows activities and include them in the schemas
Build Hyper-medias for pagination and include them in the schemas
Build a special-purpose resource for Async Tasks (/async-tasks)
Describe the API using Open API Specs and use $ref to include the representations schemas

References

The above list of rules have been collected and consolidated from the following references:

www.arcitura.com
RESTful Web APIs, by Leonard Richardson
REST API Design Rulebook, by Mark Masse
Designing Web APIs – Building APIs that developers love, by Brenda Jin
Restful Web Services Cookbook, by Subbu Allamaraju
W3C Activity Stream Vocabulary
Dublin Core
Schema.org
Microformats Wiki
http://xmlns.com/foaf/spec/
https://tools.ietf.org/html/rfc8288
https://www.iana.org/assignments/link-relations/link-relations.xhtml
https://tools.ietf.org/html/rfc6570
https://tools.ietf.org/html/rfc7807
https://tools.ietf.org/html/rfc6906
https://tools.ietf.org/html/rfc5005
https://tools.ietf.org/html/rfc4685
https://tools.ietf.org/html/rfc5829
https://tools.ietf.org/html/rfc7946
https://tools.ietf.org/html/rfc6902
https://tools.ietf.org/html/rfc3339
https://www.iso.org/iso-3166-country-codes.html
https://en.wikipedia.org/wiki/ISO_4217

Consolidated Design Rules and Standards for Great Web API

2021-02-21T15:55:00.008+03:00

In a previous thread named API Definition and History, we have explored how the APIs have evolved through the time, generation after generation until we get to the Web API generation. In this thread, we will list and consolidate the most common rules, conventions and standards for designing Great Web APIs and we will group them into the following list of categories:

Security
Performance and Reliability
Clarity and Consistency
Change Management

Security Rules

The following lists the main recommended security rules for Great API Design:

Rule 1: Prefer OAUTH over Basic Authentication

Rule 2: Define minimal Oauth Scopes

Rule 3: Isolate sensitive information in a separate scope

Rule 4: Define different scopes for different resources

Rule 5: Always verify the access token is valid and has the right permission

Make sure the access token is not expired
Make sure the access token is permitted to perform the requested HTTP method on the requested resource
If access token validation failed, an error should be returned
It is useful to return more details about the scopes needed and scopes provided. For example, return these headers:

X-OAuth-Scopes lists the scopes provided for the access token
X-Accepted-OAuth-Scopes lists the required scopes for this action

Rule 6: Issue One time use Authorization Codes

Rule 7: Issue short-lived Access Tokens

Rule 8: Support refresh tokens to allow consumers to issue new access tokens.

Rule 9: Support OAuth state parameter

Rule 10: Use SSL/TLS with mutual authentication

Rule 11: Verify provided redirect URL parameter and make sure it matches one of the registered URLs for the client application

Rule 12: Disallow rendering OAUTH page in iframes by using X-Frame-Options and/or Content-Security-Policy: frame-ancestors header

Rule 13: Use Verification Tokens with WebHook APIs

Rule 14: Use Signatures and Prevent replay attacks by including request timestamp

Rule 15: Avoid sending sensitive information as part of WebHooks

Rule 16: Provide guidelines and documentation for the API security requirements and best practices

Rule 17: Provide consumers with SDKs that implement the best security practices (authentication-using OAuth, rate-limit control, issue and validate signature, and validate request timestamp, etc.)

Performance and Reliability Rules

The following lists the main recommended performance rules for Great API Design:

Rule 18: Use Database Indexes. Note: Create indexes wisely as adding too many indexes can affect the insert, update and delete performance, for that purpose, it is recommended to follow MENTOR checklist

Rule 19: Use Database Materialized Views

Rule 20: Use Caching. Note: Remember to implement a proper invalidation and/or expiration mechanisms

Rule 21: Use Asynchronous processing for Expensive operations

Rule 22: If your API consumer uses polling to get the changes, consider event-driven APIs (such as Webhooks, websockets, and HTTP Streaming)

Rule 23: If your API is too fine-grained where few details are returned where consumer need to make multiple calls to get all the required details, then you need to introduce new coarser-grained methods to collect the frequently required details in one method (see Service API Denormalization design pattern for more details)

Rule 24: If your API is too coarse-grained where more details are returned than what they need, then you need to introduce finer-grained methods to return as minimum as required (see Lightweight Endpoint design pattern for more details).

Rule 25: If your API consumers’ requirements for the details vary, consider GraphQL API where the consumers can fetch exactly the fields they need.

Rule 26: Support Bulks; Instead of sending for bulky details as individual requests, consider supporting bulk requests where multiple entries/objects can be submitted in one request

Rule 27: Support Filtration and Ordering. If not provided the consumer may request all the data to filter it out locally which can overload the database and network.

Rule 28: If your API returns large dataset, use pagination:

Offset-based Pagination (based on Page Id and Page Size)
Cursor-based Pagination (server returns in the response, cursor for the next page which need to be submitted by the consumer if he need to get the next page)
Make sure to sort the data as per the consumer preference

Rule 29: Implement Rate-limiting strategies to control how many times an application or a client is allowed to call an API during a given time interval.

When the limit is reached, return HTTP 429 status code
Set the retry-after header to let developers programmatically retry
Make sure to document the rate limiting policies and give the consumers few best practices to not reach the rate limit

Rule 30: Encourage your consumers to use gzip compression

Rule 31: Provide consumers with SDKs that implements the best performance practices (caching, compression, pagination, rate-limit support, etc.)

Clarity and Consistency Rules

Rule 31: Enforce naming convention and standard structure for API methods, resources and payloads for better consistency.

Rule 32: Adopt standards to name your resources and their fields. The following sites define list of schemas to be consulted through the design (schema.org, GeoJSON, Microformats wiki, Dublin Core, Activity Vocabulary, FOAF)

Rule 33: Use standards for fields values. Use ISO3166 for countries, ISO4217 for currencies, RFC3339 for dates and times formats, etc.

Rule 34: Provide standard and meaningful response codes and return longer-form response codes “name_too_long”.

Rule 35: define standard schema for error reporting and use it across all the APIs responses

Rule 36: Provide human-readable error description in the response payload.

Rule 37: Provide standard form of the meta-data and a standard list of allowed meta-data fields.

Rule 38: Log errors and provide consumers a dashboard to browse and analyze the logs

Rule 39: Apply Contract-First Design approach by defining the API before start the implementation to ensure the consistency among all methods and operations

Rule 40: Pre-define the data models for your API using strict data types before starting designing any API using XML Schema and/or JSON Schema and map them with Application-specific media types.

Rule 41: Don’t use database schema as the basis for your API design.

Rule 42: Do not use Auto-generation schema tools

Rule 43: Reduce the quantity and restrictiveness of validation logic embedded in the API and keep them generic. Detailed validations in the API can make it coupled to the underlying implementation and accordingly it will be always eligible for change whenever the underlying implementation is changed. See Validation Abstraction design pattern for more details

Rule 44: Pre-define problem detail schema and use it across all the APIs

Rule 45: Support multiple formats (JSON, XML, etc.) and allow the consumer to negotiate for his preferred format (see Content Negotiation design pattern for more details)

Rule 46: Use Open API Specification to describe your API and use $ref to reference the pre-defined data model Schema (See Rule#40&44) that increases the reusability and improves the consistency

Rule 47: Always validate requests against the defined schema and respond back with meaningful error code and message in case of failure. As per Rule#43 keep the schema constraints to the minimum

Rule 48: You must use the same format for both success and failure responses. For example If you are using JSON for success response, then you must use JSON for failure response

Rule 49: Provide documentation to help developers get started

Provide tutorials and Getting start guides
Provide examples for request and response payload
Provide sample code
Provide API technical specifications (see Rule#46)
Provide best practices guidelines and terms of service
Provide FAQ

Rule 50: Provide interactive documentation online, where developers can test out the API

Rule 51: Provide consumers with SDKs that implements the best practices (error handling, retry for rate-limit errors, perform OAuth authentication, pagination, etc.)

Change Management Rules

Rule 52: Make sure your changes are consistent and following your standard naming convention, error coding, logging, etc.

Rule 53: Set up versioning system and standard versioning scheme from the beginning (see Canonical Versioning design pattern for more details) and use Version Identification design pattern through the API design

Recommended versioning scheme (MAJOR.MINOR.PATCH)

MAJOR for backward incompatible changes (See Rule#55)
MINOR for backward compatible changes (See Rule#54)
PATCH for backward compatible bug fixes

Another recommended versioning scheme is to apply the backward compatible changes seamlessly without changing the version number (e.g., v1, v2, v3, etc.)

Rule 54: The following changes can be considered backward compatible and can be applied at any time with a proper communication:

Adding new Operation, new method or new endpoint
Adding optional field to the request payload
Deleting optional fields from the request or response (Unless the response payload is expected to change based on this request field)
Loosen request field restrictions. For example: request field max length is increased from 10 to 20 or changing integer data type to string
Tighten response field restrictions. For example: response field max length is decreased from 20 to 10 or changing string data type to integer

Rule 55: The following changes are not backward compatible and accordingly proper change management techniques should be applied to minimize the impact on the consumers:

Deleting or renaming operation, method or endpoint
Add Mandatory field to the request payload
Add Field to the response payload
Delete or rename mandatory field from the request payload
Delete or rename mandatory field from the response payload
Change the order of the fields in case of XML schema that doesn’t use <xsd:all>. In case of JSON, the properties are, by default, unordered.
Tighten request field restrictions. For example: request field max length decreased from 20 to 10 or changing string data type to integer
Loosen response field restrictions. For example: response field max length increased from 10 to 20 or changing integer data type to string
Changes on error codes and error codes meaning
Change the default data format (e.g., changing the default from json to xml or vice versa)

Rule 56: Apply the following techniques to minimize the impact on the consumer in case of backward incompatible changes:

Utilize the existing versioning system and versioning scheme to introduce a new version (See Rule#2)
Introduce Service API proxy design pattern for existing customers. API proxy retains the original API and contains logic to accommodate the changes.
In case of endpoint change, apply Endpoint Redirection design pattern using Location header to redirect the consumer to the new endpoint
Apply Termination Notification design pattern to announce version has been deprecated and announce the date of old version retirement
Build change log to track the versions and changes applied in each version
Once the old service is switched service, server should return status code (410 “Gone”)

Rule 57: Encourage your clients:

To not fail if they received a new field in the response
To support unordered fields for the data models in the request and response

References

The above list of rules have been collected and consolidated from the following references:

www.arcitura.com
RESTful Web APIs, by Leonard Richardson
REST API Design Rulebook, by Mark Masse
Designing Web APIs – Building APIs that developers love, by Brenda Jin
Restful Web Services Cookbook, by Subbu Allamaraju
W3C Activity Stream Vocabulary
Dublin Core
Schema.org
Microformats Wiki
http://xmlns.com/foaf/spec/
https://tools.ietf.org/html/rfc7807
https://tools.ietf.org/html/rfc3339
https://www.iso.org/iso-3166-country-codes.html
https://en.wikipedia.org/wiki/ISO_4217

OAuth and OpenID Connect Clarifications

2021-02-15T13:37:00.003+03:00

Whenever developers start working with SSO and whenever they need to build a secure API, they usually raise questions like:

What is the difference between Open ID Connect and OAUTH?
When to use Open ID Connect and when to use OAUTH?
What is the difference between Open ID Connect and SAML?
For APIs, why do I need OAUTH? Can I just use basic authentication?

In this thread, I will try to list the different possible use cases, simplify the concepts and explain the different possible flows in detail to eliminate any confusion.

OAuth in nutshell

In the following, we will discuss OAUTH version 2.0 which is the de-facto

OAUTH is the base and we must understand how OAUTH works before going to Open ID Connect. It will be confusing for you, if you tried to understand Open ID Connect before understanding OAUTH.

At a high level, OAUTH is a generic standard framework that controls the delegation and authorization to a protected resource.

But what protected resource mean?

The resource can be anything that can be used for a purpose. For example, the file, record in a database, or even a mathematical calculation can be considered as resources. The resources are usually accessible (exposed) through an API or a webservice. If the resource needs to be protected from unauthorized access, then it is considered as protected resource. For example, my details in Facebook is considered as protected resources because Facebook is not supposed to share my details with any other application without my permission.

According to the above definitions, OAUTH allows end-users to give (delegate) to 3^rd-party client applications (we will call it later “Client Application”) the permission to access their protected resources. For example, OAUTH on Facebook allows me, as end-user, to give Twitter application the permission to post on my Facebook page on my behalf.

OAUTH manages the resources by assigning them scopes. Each scope specifies which resource can be accessed and can also specify the type of access (read or write). OAUTH gives you, as OAUTH server owner, the flexibility to define the scopes you want the way you want it, the following are just examples for possible scopes:

Read_profile
post_thread
Write_profile

OAUTH supports 3 main authorization scenarios (They call them flows):

Authorization Code Flow which is suitable for Web-Applications that have end-user involved
Client Credentials Flow which is suitable for server-to-server communication and there is no end-user involved
Resource Owner Password Flow (it is not recommended and accordingly I will not explain it in this thread)

In the above flows, OAUTH represents the permission granted using a temporarily code called Authorization Code which allows this specific Client Application to access this specific scope on behalf of this specific end-user and can’t be used by any other user or any other Client Application.

Once the authorization code is granted to the Client Application, then the application can request for Access Token which can be used to call the API.

In general, no matter what is the flow we are using, Client Application must aim to get the authorization code (the granted permission) which allows it to get the access token that can be used for the API invocation.

As per the above diagram, Client Application must follow the below high-level procedure:

Step 1: Client Application need to get the authorization code

This step is only required for Authorization Code Flow, authorization code can be retrieved by redirecting the end-user to OAUTH server on URI /authorize

Step 2: Get the access token based on the retrieved authorization code

This is the same for both flows
The access token can be retrieved by calling the following URI /oauth/token by passing either client id/client secret or by passing the authorization code depends on the flow

Step 3: Then you can call the API using access token.

Access token has an expiration that can last for several minutes, hours or even days depending on OAUTH server configuration
Client Application can keep using the access token for multiple API calls as long as it is not expired

According to the above, you may ask what is the value of using OAUTH Client Credentials flow? Can I just use Basic Authentication?

OAUTH Client Credentials flow provides two main benefits:

First benefit is related to API design. If your API need to support two types of authorizations; one where an end-user involved and another one with your trusted partners where no end-user is involved, with Basic Authentication you need to provide two versions of the API: The first version should support access token for authorizations that have end-uses involved, and the second version should support Basic Authentication for trusted Client Applications where no end-user is involved.

If OAUTH is used for both cases, your API design will be consistent as it will always expect access token regardless if there is an end-user involved or not.

The second benefit is related to the API security. With Basic Authentication, the Client Application is required to send the Client Secret with every single request which can be tracked and exposed by the hackers. With OAUTH, if the access token is breached, it will be much easier to revoke it and generate a new one without the need to revoke the client secret itself.

OpenID Connect in nutshell

It is important to note than we are talking here about OpenID Connect not OpenID 1 nor OpenID 2 which are deprecated.

OpenID Connect (OIDC) is an implementation on top of OAUTH to get end-user details to support SSO and authentication. You can think with OIDC as a backend API that allows the Client Application to retrieve the end-user profile if the end-user give the permission.

As explained before OAUTH consists from 4 main concepts:

Protected resource
Backend API that exposed the protected resource
Scopes
Authorization Flows

With OpenID Connect, the above concepts have been standardized to support the user identification and authentication as following:

With OIDC, the Protect resource is the end-user details and it is standardized using especial purpose JWT formatted Token called “ID Token”. ID Tokens are signed using the server private key and Client Application can validate the ID Token using server public key which can be retrieved from” /oidc/.well-known/openid-configuration
With OIDC, the Backend API is the API that Client Application can call to get the end user details using URL /userinfo
With OIDC, the Scopes are standardized to following 4 scopes:

openid; this is used to validate the user’s identity without retrieving any data
profile; this is used to get user name and any other available details
email; this is used to get user’s email

With OIDC, the Authorization Code Flow has been extended into the following 3 flows as depicted in the below diagram

As per the above flows, OIDC uses OAUTH to permit Client Applications to get the user info that is required for SSO and authentication (and it may also be used for self-registration).

SAML at a high-level

Now, it is the time to answer the question: what are the differences between SAML and OIDC? Both are standards for user identification and authentication and the following are the main high-level differences:

	OIDC	SAML
Support Mobile Application	Yes	No
Simplicity	Simple	Complex
Data Format	JSON & JWT	XML

Summary

OAUTH is meant for users or applications authorization, OIDC is build on top of OAUTH to support user identification and authentication and SAML is another standard for user identification and authentication.

If you are an API developer, you should focus on OAUTH standard and accordingly, your API should be built to authorize the requests based on the provided access token.

API Definition and API History

2021-02-11T15:58:00.002+03:00

Introduction

One of the most confusing questions I usually have is "What are the differences between API and webservices"?

In order to answer this question we need to understand first what API means?

Definition

API stands for Application Programming Interface and as its name implies, it is an interface that allows any piece of code to interact programmatically with another piece of code. The piece of code can be library, component, resource or even a different application.

The APIs can be an internal interface that allowing native interactions such as calling a reusable library or component within the same application and with the same deployment environment. While the external APIs allow remote interactions such as calling another component for the same application deployed on a different environment in a distributed application architecture or even calling another application. The external interactions require a network communication supported by a network protocol (e.g., HTTP, JMS, TCP, etc.).

Well designed APIs (internal or external), can increase the solution reusability, hide implementation complexity, and decreases the clients dependencies on the implementation which allows the implementation to evolve Independently without affecting the clients.

In the below diagram, the java interface called "SearchI" is considered an API that allows the client code to interact with the searching implementation without getting coupled to the underling implementation:

In the above diagram, if we decided to change the BinarySearch class with LinearSearch, the client will not be affected as long as the interface specifications are not changed.

APIs usually consists from three components:

The activities that can be performed by the underling implementation

An example for internal API activities is the list of methods defined in the java interface
An example for external API activities is the list of operations defined in a webservice WSDL

The data model that describes the entities and attributes need to be exchanged with the interface

An example for data model is CompanyProfile entity that need to be passed to the API to perform a specific activity

The data format that describes the shape of the data

For internal APIs, the data format can be as per the programming language native formats
For external APIs, the data format can be SOAP, JSON, or even serialized binary format such as gRPC.

API Generations

API went through 4 generations as illustrated in the below timeline:

As depicted by the above diagram, the REST-based and SOAP-based Webservices are considered as styles of the 3rd generation APIs where remote API calls can be performed over internet using mainly HTTP protocol.

In the following diagram, we will zoom in the Web API generation and see how the Web API has evolved:

Conclusion

It is clear now that asking about the differences between APIs and Webservices is not valid and the right question is supposed to be "which type of APIs is the webservices?".

Microservices Design Patterns (Arcitura vs Microservices.io vs Microsoft)

2021-01-17T13:11:00.019+03:00

In the market today, there are 3 schools for Microservices Design Patterns; Microsoft, Microservices.io and Arcitura.

The three schools propose 86 different design patterns, with different names, providing solutions to achieve the Microservices drivers (Agility and Scalability).

Total Design Patterns provided by Arcitura is 41 Design Patterns (11 of them are inherited from SOA Design Patterns), 12 of them are unique for Arcitura.
Total Design Patterns provided by Microservices.io is 47 Design Patterns, 20 of them are unique for Microservices.io
Total Design Patterns provided by Microsoft is 37 design pattern, 16 of them are unique for Microsoft
Only 24 design patterns are shared between Arcitura and Microservices.io (Note: 2 design patterns from Arcitura are mapped to 5 design patterns from Microservices.io)
Only 6 design patterns are shared between Arcitura and Microsoft
Only 2 design patterns are shared between Microsoft and Microservices.io
Only 16 design patterns are shared between Arcitura and Microservices.io
Only 13 design patterns are shared between the 3 schools (highlighted in gray)

Arcitura and Microsoft focus more on the Deployment, Performance, Scalability and Reliability Patterns, while Microservices.io focus more on the Decomposition, Development, Testing, Traceability and Transaction processing patterns.

For people looking for agility, it would be recommended for them to refer to Microservices.io

For people looking for performance, scalability and reliability, it would be recommended for them to refer to Arcitura

The below table maps the the different design patterns from the three schools with the related products and technologies

Note: Design pattern named CQRS in Microsoft is different than the one in Microservices.io

#	Microsoft	Arcitura	Microservices.io	Related Product/ Tech
	Decomposition & Layering
	NA	1. Micro Task Abstraction	NA	NA
	1. CQRS	2. Micro Task Segregation	NA	NA
	NA	NA	1. Decompose by business capability	NA
	NA	NA	2. Decompose by subdomain	NA
	NA	NA	3. Service per Team	NA
	NA	3. Microservice Layers	NA	NA
	2. Strangler Fig	NA	NA
	Deployment
	3. Deployment stamps	4. Composition Autonomy (SOA)	4. Self-Contained Service	Docker Compose
9.	NA	5. Microservice Deployment	5. Service deployment platform	Docker Kubernetes
10.	4. Compute Resource Consolidation	6. Single Node Multi-Containers	6. Multiple service instances per host	Docker
11.	NA	7. Rich Container	NA	Docker Kubernetes
12.	NA	8. Logical Pod Container	NA	Kubernetes Pod Openshift Pod
13.	NA	9. Serverless Microservice Deployment	7. Serverless deployment	AWS Lambda Google Cloud Functions Azure Functions
14.	NA	10. Multi-Container Isolation Control	NA	Openshift Pod
15.	NA	11. Microservice Isolation Levels	8. Service instance per host
16.	NA		9. Service instance per VM	VMWare, vSphere, Amazon AWS
17.	NA		10. Service instance per Container	Docker Kubernetes
	Data Management & Data Interchange
18	NA	12. Dedicated Microservice Database Service Data Replication (SOA)	11. Database per service	Any Database (MySQL, Oracle)
19	NA	13. Shared Isolated Database	12. Shared Database	Any Database (MySQL, Oracle)
20	5. Materialized View	14. Pre-Defined Data View	13. CQRS	Any Database (MySQL, Oracle) Eventuate
21	6. Gateway Aggregation	15. Microservice Compositor	14. API Composition	Compoxure Netflix Zuul
22	7. Gateway Routing	16. Autonomous Proxy Service	15. API Gateway	NGINX Netflix Zuul
23	NA	17. Centralized Isolated State Repository	NA	Apache Zookeeper
24	NA	18. Synchronized Cross-Instance Events	16. Domain Event	Azure Cosmos Eventuate
25	8. Sharding	NA	NA
	Transactional Messaging
26	NA	NA	17. Transactional outbox
27	NA	NA	18. Transaction log tailing	Eventuate Tram
28	NA	NA	19. Polling publisher	Eventuate Tram
29	9. Compensating Transaction	19. Compensating Service Transaction	20. Saga	Eventuate
	Integration & Event Processing
30	10. Sidecar	20. Container sidecar	NA	AirBnB SmartStack
31	11. Ambassador	21. Microservice Ambassador	NA	Docker Ambassador
32	NA	22. Service Messaging (SOA)	21. Remote Procedure Invocation	Rest gRPC Apache Thrift
33	12. Competing Consumers	23. Async Queuing (SOA)	22. Async Messaging	Apache Kafka RabbitMQ Anypoint MQ
34	13. Queue-Based Load Leveling
35	14. Async Request-Reply
36	15. Sequential Convoy
37	NA	24. Canonical Protocol (SOA)	23. Domain-specific Protocol	HTTP SMTP IMAP
38	NA	25. Idempotent Capability (SOA)	24. Idempotent Consumer
39	16. Event Sourcing	26. Appended Events	25. Event Sourcing	Kafka MapR Streams Eventuate
40	NA	27. Atomic Event Processing	NA	Anypoint MQ
41	17. Publisher/Subscriber	28. Event-Driven Messaging (SOA)	NA	Anypoint MQ
42	18. Choreography	NA	NA
43	19. Anti-Corruption Layer	NA	NA
44	20. Pipes and Filters	NA	NA
45	21. Scheduler Agent Supervisor	NA	NA
	Registry and Discovery
46	NA	29. Metadata Centralization	26. Service registry
47	NA	30. Microservice Instance Registration	27. Self Registration	Netflix Eureka
48	NA	30. Microservice Instance Registration	28. 3rd Party Registration	Netflix Prana AWS Autoscaling Groups Registrator
49	NA	NA	29. Client-side discovery
50	NA	NA	30. Server-side discovery
	Scalability & Reliability
51	NA	31. Dynamic Scalability	NA	Amazon AWS Dynamic Scaling
52	NA	32. Service Load Balancing	NA	Amazon Elastic F5 Load Balancer
53	NA	33. Workload Distribution	NA	VMWare DRS
54	NA	34. Container Chain	NA	Kubernetes Init Container
55	22. Leader Election	35. Leader Node Election	NA	ZooKeeper
56	NA	36. Micro Scatter-Gather	NA	Apache Camel Spring Integration
57	23. Backends for Frontends	37. Redundant Implementation (SOA)	NA	Amazon AWS Docker
58	NA	38. Volatile Configuration	NA	Docker
59	24. Circuit Breaker	NA	31. Circuit Breaker
60	25. Bulkhead	NA	NA
61	26. Cache-Aside	NA	NA
62	27. Claim-Check	NA	NA
63	28. Geodes	NA	NA
64	29. Retry	NA	NA
65	30. Index Table	NA	NA
66	31. Throttling	NA	NA
	Utility
67	NA	NA	32. Microservice chassis	Spring Boot DropWizard
68	NA	NA	33. Service Template
69	32. External Configuration Store	NA	34. Externalized configuration
70	33. Gateway Offloading	NA	NA
71	34. Health Endpoint Monitoring	NA	NA
	Traceability
72	NA	39. Distributed Diagnostics	35. Distributed tracing	QBit LMDC Zipkin
73	NA	40. Log Aggregator	36. Log aggregation	Loggy Container AWS Cloud Watch
74	NA	NA	37. Application metrics	Prometheus AWS Cloud Watch
75	NA	NA	38. Audit logging
76	NA	NA	39. Exception tracking
77	NA	NA	40. Health check API
78	NA	NA	41. Log deployments and changes	AWS Cloud Trail
	Testing
79	NA	NA	42. Service Component Test	Spring Cloud Contract
80	NA	NA	43. Consumer-driven contract test	Spring Cloud Contract
81	NA	NA	44. Consumer-side contract test
	Security
82	35.Federated Identity	41. Brokered Authentication (SOA)	45. Access Token
83	36.Valet Key	41. Brokered Authentication (SOA)	45. Access Token
84	37.Gatekeeper	NA	NA
	UI Patterns
85		NA	46. Server-side page fragment Composition
86		NA	47. Client-side UI Composition

Enterprise Integration Patterns (Table)

2019-07-26T03:41:00.000+03:00

Enterprise Integration Patterns	Integration Style	Root Pattern	Base Pattern	IIB Implementation Reference
	File Transfer
	Shared Database
	Remote Procedure Call (RPC)
	Messaging For Loosely Coupled Integration, Messaging System (Middleware) is needed and it must support following concepts	Message
			Command Message		Command Message Pattern in IIB
			Document Message
			Event Message
			Request-Reply
			Return Address
			Correlation Identifier
			Message Sequence
			Message Expiration
			Format Indicator
		Message Channel (Async)
			Point-to-Point
			Publish-Subscribe
			Datatype Channel
			Invalid Message Channel
			Dead Letter
			Guaranteed Delivery
			Channel Adapter
			Message Bridge
			Message Bus
		Pipes and Filters
		Message Router
			Content-Based Router
			Message Filter
			Dynamic-Router
			Recipient List
			Splitter
			Aggregator
			Re-sequencer
			Composed Message Processor
			Scatter-Gather
			Routing Slip
			Process Manager
			Message Broker
		Message Translator
			Envelope Wrapper
			Content Enricher
			Content Filter
			Claim Check
			Normalizer
			Canonical Data Model
		Message Endpoint
			Messaging Gateway
			Messaging Mapper
			Transactional Client
			Polling Consumer
			Event-Driven Consumer
			Competing Consumers
			Message Dispatcher
			Selective Consumer
			Durable Subscriber
			Idempotent Receiver
			Service Activator

Desktop MVC vs Web MVC

2019-07-24T15:31:00.000+03:00

MVC is a pattern in software design commonly used to implement user interfaces, data, and controlling logic. It emphasizes a separation between the software’s business logic and display. The below diagram depicts how MVC is implemented:

From the above diagram you can see how the model updates the views directly (using Observer design pattern) and that's because the original MVC was designed mainly for desktop applications where the model and view are just objects on the same machine, but the story is different in the web applications.
In Web applications, the model resides on the server and the view is just a set of html pages rendered on users browser and there is no direct contact between the model and users' browser. For that reason, the MVC has been evolved to adapt the web applications by giving the controller more responsibility. below diagram depicts the Web MVC:

In Web MVC, the controller accepts the user input, manipulates the model and sends/renders the updates to the views.

Different Types of Technology Architectures and their relationships with Design Patterns

2018-07-20T12:46:00.017+03:00

The following diagram shows the different types of Technology Architectures and the relationship between them:

In the following, you will find the most common definition for each Software Architecture type starting from the parent (i.e., Technology Architecture):
Technology Architecture
The fundamental and foundational aspects of physical design of some piece of technology. SOA Design Patterns by Erl
Software Architecture

Software Architecture describes the fundamental structure of a software.
Software Architecture can be either Application Architecture, Integration Architecture, or Data Architecture and it can mean all of them.
Application Architecture
Application Architecture is a description of the subsystems and components of a software and the relationships between them. Subsystems and components are typically detailed in different views to show the relevant functional and non-functional properties of a software. A component is an encapsulated part of a software system. A component has an interface. Components serve as the building blocks for the structure of a system. At a programming-language level, components may be represented as modules, classes, objects or a set of related functions.
Note: Sometimes people say Software Architecture and they mean Application Architecture.
Pattern-Oriented Software Architecture, Volume 1, SOA Design Patterns by Erl
Integration Architecture
The description of two or more connected applications including whatever technologies, resources, or extensions were added to enable their integration.
Note: Integration Architecture is different from Distributed Application Architecture. Distributed Architecture parts are communicating with each other in a tightly coupled manner (one tier cannot work without the other); the communication between the parts must be synchronous. In contrast, Integrated Applications are independent applications that can each run by themselves but that coordinate with each other in a loosely coupled way.
SOA Design Patterns by Erl, Enterprise Integration Patterns by Hophe
Data Architecture
Data Architecture is composed of models, policies, rules or standards that govern which data is collected, and how it is stored, arranged, integrated, and put to use in data systems and in organizations. TOGAF® 9.1 - Phase C: Information Systems Architectures - Data Architecture
Enterprise Application Architecture
Enterprise Applications is a sub-type of Application Architecture that requires involvement of persistent data and it must support concurrent access through many UI screens. Enterprise Applications are also known as Information Systems.
Enterprise Application can be structured or distributed. A distributed application is an application whose components are located on different machines communicating with each other synchronously. For example, Client/Server architecture. Patterns of Enterprise Application Architecture
Solution / System Architecture
Solution Architecture is a description of an entire system in the context of a solution that is related to specific business problems. Solution Architecture should describe Software and Hardware Architectures of the entire solution.
Enterprise Architecture
Enterprise Architecture applies architecture principles and practices to guide organizations through the business, information, process, and technology changes necessary to execute their strategies. These practices utilize the various aspects of an enterprise to identify, motivate, and achieve these changes. Federation of EA Professional Organizations, "Common Perspectives on Enterprise Architecture
In the following table, the main Software-related Architectural patterns are listed along with their relationships with the different Design Patterns:

Software Patterns
Architectural Patterns vs Design Patterns

Architectural Patterns

(Common Architectural solutions for Common Architectural problems)

Architectural pattern expresses a fundamental structural organization schema for software systems. It provides a set of predefined subsystems, specifies their responsibilities, and includes rules and guidelines for organizing the relationships between them.

Below is a list of examples:

Design Patterns

(Common Design solutions for Common Design problems)

Design Pattern provides a scheme for refining the subsystems or components of a software system, or the relationships between them. It describes a commonly recurring structure of communicating components that’s solves a general design problem within a particular context.

Below is a list of examples:

Structured Application Architecture

* MVC

* Multi-Layered

OO Design Patterns (Can be applied within structured or distributed applications components/services built using OOP)

Examples:

Decorator

Builder

* Observer

* Façade

* Command

Distributed Application Architecture

* N-Tier

* SOA

* Microservices

Integration Architecture

* Hub and Spoke

* Pipes and Filters

Integration Design Patterns

Find Enterprise Integration Patterns blog for more details

Design Patterns specific for particular Architectural Patterns

For each sub-type of the architectural patterns, there are a set of related design patterns: Enterprise Application Architecture Design Patterns, SOA Design Patterns, Microservices Design Patterns, etc.

Concurrency Design Patterns (Can be applied to any Architecture)

Examples:

* Thread Pool

* Scheduler

* Lock

Security Design Patterns (Can be applied to any Architecture)

Examples

* Authenticator

* Input Validation

Apache Mina: Client side to Correlate Response with its Request

2013-05-27T13:48:00.002+03:00

It is very common to build a TCP client by starting a thread for each request and each request opens its own TCP channel.

In this blog, we will explain how to build a client that maintains one channel to handle all requests and all responses. This model is required for low-latency solutions.

Because one channel will be used for all messages then each message should have two extra fields:

. Message size: this field is important to determine when the message end and when the next message begins
. Correlation Id: this Id is important to match the response with its request

This blog is dependent on Apache Mina Framework. Where the following classes will be used:

Transaction: this class is required to store the request and its matching response
TransactionDecoder: this is required to convert the IO Buffer into Transaction object.
TransactionEncoder: this is required to convert the Transaction object into IO Buffer.
Multiplexer: This class is required to send a request and wait for response
ClientSessionHandler: this is the handler required by Apache Mina.

We will start with the Transaction class:

package com;

import java.util.concurrent.ArrayBlockingQueue;

import java.util.concurrent.BlockingQueue;

import java.util.concurrent.TimeUnit;

import java.util.concurrent.atomic.AtomicInteger;

public class Transaction {

// This object will be used as sequencer to generate new correlation ids

public static AtomicInteger messageIdSeq = new AtomicInteger();

private int correlationId;

private String request;

private String response;

private final BlockingQueue availableResponse = new ArrayBlockingQueue(1);

public Transaction() {

// For each new transaction, a new correlation Id will be created

this.correlationId = messageIdSeq.incrementAndGet();

}

public String getResponse() {

return this.response;

}

public String getRequest() {

return this.request;

}

public int getCorrelationId() {

return correlationId;

}

public void setResponse(String response) {

this.response = response;

}

public void setRequest(String request) {

this.request = request;

}

public void setCorrelationId(int correlationId) {

this.correlationId = correlationId;

}

public void responseReceived(String response) {

// This method will be called to announce the response arrival

this.response = response;

availableResponse.offer(response);

}

public String waitResponse(long timeout, TimeUnit unit) throws InterruptedException {

// this method will make the caller to wait until a response is received

return availableResponse.poll(timeout, unit);

}

@Override

public String toString() {

return "-->"+correlationId+"-"+request+"-"+response;

}

Here is the Encoder:

package com;

import java.nio.charset.Charset;

import java.nio.charset.CharsetEncoder;

import org.apache.mina.core.buffer.IoBuffer;

import org.apache.mina.core.session.AttributeKey;

import org.apache.mina.core.session.IoSession;

import org.apache.mina.filter.codec.ProtocolEncoderAdapter;

import org.apache.mina.filter.codec.ProtocolEncoderOutput;

public class TransactionEncoder extends ProtocolEncoderAdapter {

private final AttributeKey ENCODER = new AttributeKey(getClass(), "encoder");

private Charset charset;

private boolean isClient;

public TransactionEncoder(Charset charset, boolean isClient) {

this.charset = charset;

this.isClient = isClient;

}

@Override

public void encode(IoSession session, Object message, ProtocolEncoderOutput out) throws Exception {

CharsetEncoder encoder = (CharsetEncoder) session.getAttribute(ENCODER);

if (encoder == null) {

encoder = charset.newEncoder();

session.setAttribute(ENCODER, encoder);

}

Transaction trxn = (Transaction) message;

IoBuffer buffer = IoBuffer.allocate(8+trxn.getRequest().length()).setAutoExpand(true);

if (isClient) {

// Add request size as header

buffer.putInt(4+trxn.getRequest().length());

// Add correlation Id

buffer.putInt(trxn.getCorrelationId());

// Add message

buffer.putString(trxn.getRequest(), encoder);

} else {

// Add response size as header

buffer.putInt(4+trxn.getResponse().getBytes(charset).length);

// Add correlation Id

buffer.putInt(trxn.getCorrelationId());

// Add message

buffer.putString(trxn.getResponse(), encoder);

}

buffer.flip();

out.write(buffer);

}

Here is the Decoder:

package com;

import java.nio.ByteBuffer;

import java.nio.CharBuffer;

import java.nio.charset.Charset;

import java.nio.charset.CharsetDecoder;

import org.apache.mina.core.buffer.IoBuffer;

import org.apache.mina.core.session.AttributeKey;

import org.apache.mina.core.session.IoSession;

import org.apache.mina.filter.codec.CumulativeProtocolDecoder;

import org.apache.mina.filter.codec.ProtocolDecoderOutput;

public class TransactionDecoder extends CumulativeProtocolDecoder {

private final AttributeKey DECODER = new AttributeKey(getClass(), "decoder");

private Charset charset;

private boolean isClient;

public TransactionDecoder(Charset charset, boolean isClient) {

this.charset = charset;

this.isClient = isClient;

}

@Override

protected boolean doDecode(IoSession session, IoBuffer in, ProtocolDecoderOutput out) throws Exception {

CharsetDecoder decoder = (CharsetDecoder) session.getAttribute(DECODER);

if (decoder == null) {

decoder = charset.newDecoder();

session.setAttribute(DECODER, decoder);

}

if (in.prefixedDataAvailable(4)) { // first 4 bytes are required to get the length

int length = in.getInt();

int correlationId = in.getInt();

length = length - 4;

if (isClient) {

Transaction trxn = new Transaction();

trxn.setCorrelationId(correlationId);

// Read response message

byte[] bytes = new byte[length];

in.get(bytes);

CharBuffer buffer = decoder.decode(ByteBuffer.wrap(bytes));

trxn.setResponse(new String(buffer.array()));

out.write(trxn);

} else {

// Read request message

byte[] bytes = new byte[length];

in.get(bytes);

CharBuffer buffer = decoder.decode(ByteBuffer.wrap(bytes));

Transaction trxn = new Transaction();

trxn.setCorrelationId(correlationId);

trxn.setRequest(new String(buffer.array()));

out.write(trxn);

}

return true;

}

return false;

}

Apache Mina requires a factory class for the decoder and encoder:

package com;

import java.nio.charset.Charset;

import org.apache.mina.core.session.IoSession;

import org.apache.mina.filter.codec.ProtocolCodecFactory;

import org.apache.mina.filter.codec.ProtocolDecoder;

import org.apache.mina.filter.codec.ProtocolEncoder;

public class TransactionCodecFactory implements ProtocolCodecFactory {

private ProtocolEncoder encoder;

private ProtocolDecoder decoder;

public TransactionCodecFactory(Charset charset, boolean isClient) {

encoder = new TransactionEncoder(charset, isClient);

decoder = new TransactionDecoder(charset, isClient);

}

public ProtocolEncoder getEncoder(IoSession ioSession) throws Exception {

return encoder;

}

public ProtocolDecoder getDecoder(IoSession ioSession) throws Exception {

return decoder;

}

Here is the Multiplexer who will send the request and waits for response and uses a ConcurrentMap to map the transaction id (correlation id) with its transaction.

package com;

import java.util.concurrent.ConcurrentHashMap;

import java.util.concurrent.ConcurrentMap;

import java.util.concurrent.TimeUnit;

import org.apache.mina.core.future.WriteFuture;

import org.apache.mina.core.session.IoSession;

public class Multiplexer {

private static ConcurrentMap responseMap = new ConcurrentHashMap();

public static Transaction sendAndReceive(IoSession session, String request) throws Throwable {

Transaction trxn = new Transaction();

trxn.setRequest(request);

responseMap.put(trxn.getCorrelationId(), trxn);

// Send the request

WriteFuture writeFuture = session.write(trxn);

// Send and wait for response

try {

boolean isSent = writeFuture.await(30, TimeUnit.SECONDS);

if (!isSent) {

// timeout

throw new Exception("Could not sent the request in " + 30 + " seconds.");

}

if (writeFuture.getException() != null) {

throw writeFuture.getException();

}

trxn.waitResponse(30, TimeUnit.SECONDS);

responseMap.remove(trxn.getCorrelationId());

if (trxn.getResponse() == null) {

// timeout

throw new Exception("Could not receive the response in " + 30 + " seconds.");

}

return trxn;

} catch (InterruptedException e) {

throw e;

} catch (Exception e) {

throw e;

} catch (Throwable e) {

throw e;

}

public static void resposeReceived(Transaction trxn) {

if (responseMap.get(trxn.getCorrelationId()) != null) {

responseMap.get(trxn.getCorrelationId()).responseReceived(trxn.getResponse());

} else {

System.out.println("Multiplexer.resposeReceived()>>>This response is already timedout:"+trxn.getCorrelationId());

}

Now we will show you the Handler; it is a simple class where it will only notify the multiplexer that a response is a received and the Multiplexer will do the needed correlation based on its ConcurrentMap and the correlation Id

package com;

import org.apache.mina.core.service.IoHandler;

import org.apache.mina.core.session.IdleStatus;

import org.apache.mina.core.session.IoSession;

import org.apache.mina.transport.socket.nio.NioSocketConnector;

public class ClientSessionHandler implements IoHandler {

// private NioSocketConnector connector;

public ClientSessionHandler(NioSocketConnector connector) {

// this.connector = connector;

}

@Override

public void exceptionCaught(IoSession arg0, Throwable arg1) throws Exception {

arg1.printStackTrace();

}

@Override

public void messageReceived(IoSession arg0, Object o) throws Exception {

Transaction trxn = (Transaction) o;

Multiplexer.resposeReceived(trxn);

}

@Override

public void messageSent(IoSession session, Object message) throws Exception {

}

@Override

public void sessionClosed(IoSession arg0) throws Exception {

reconnect(arg0);

}

private void reconnect(IoSession session) {

// Wait for five seconds before reconnecting.s

// try {

// Thread.sleep( 5000 );

// } catch (InterruptedException e) {

// e.printStackTrace();

// }

// Reconnect.

// connector.connect( session.getRemoteAddress());

}

@Override

public void sessionCreated(IoSession arg0) throws Exception {

System.out.println("ClientSessionHandler.sessionCreated(): "+arg0.getId());

}

@Override

public void sessionIdle(IoSession arg0, IdleStatus arg1) throws Exception {

System.out.println("ClientSessionHandler.sessionIdle(): "+arg0.getId());

}

@Override

public void sessionOpened(IoSession arg0) throws Exception {

System.out.println("ClientSessionHandler.sessionOpened(): "+arg0.getId());

}

Here is a test client:

package com;

import java.net.InetSocketAddress;

import java.nio.charset.Charset;

import java.util.concurrent.Callable;

import java.util.concurrent.CompletionService;

import java.util.concurrent.ExecutorCompletionService;

import java.util.concurrent.ExecutorService;

import java.util.concurrent.Executors;

import javax.swing.JOptionPane;

import org.apache.mina.core.future.CloseFuture;

import org.apache.mina.core.future.ConnectFuture;

import org.apache.mina.core.future.IoFuture;

import org.apache.mina.core.future.IoFutureListener;

import org.apache.mina.core.session.IoSession;

import org.apache.mina.filter.codec.ProtocolCodecFilter;

import org.apache.mina.filter.logging.LoggingFilter;

import org.apache.mina.transport.socket.SocketSessionConfig;

import org.apache.mina.transport.socket.nio.NioSocketConnector;

public class MainClient {

public static void main(String[] args) throws Exception {

IoSession session = init();

write(session, "HELLO SERVER")

}

private static Transaction write(final IoSession session, final String request) {

Transaction trxn = null;

try {

trxn = Multiplexer.sendAndReceive(session, request);

System.out.println("XXX:"+trxn.getCorrelationId()+"==>Response:"+trxn.getResponse()+", trxn.request:"+trxn.getRequest());

} catch (InterruptedException e) {

e.printStackTrace();

} catch (Exception e) {

e.printStackTrace();

} catch (Throwable e) {

e.printStackTrace();

}

return trxn;

}

private static IoSession init() throws Exception {

NioSocketConnector connector = new NioSocketConnector();

connector.setConnectTimeoutMillis(30000);

((SocketSessionConfig) connector.getSessionConfig()).setTcpNoDelay(true);

connector.getFilterChain().addLast("plain_logger", new LoggingFilter());

connector.getFilterChain().addLast("codec", new ProtocolCodecFilter(new TransactionCodecFactory(Charset.forName("UTF-8"), true)));

connector.getFilterChain().addLast("encoded_logger", new LoggingFilter());

connector.setHandler(new ClientSessionHandler(connector));

ConnectFuture connectFuture = connector.connect(new InetSocketAddress("127.0.0.1", 2345));

IoSession session = connectFuture.await().getSession();

return session;

}

Not Only RDBMS

2013-03-30T03:35:00.000+03:00

Since the 1970’s, relational databases became the mainstream due to several reasons:

Simplicity.
Easy ad-hoc-queries, with very powerful query languages.
Solid normalization rules.
Support JOIN operations between multiple tables through foreign keys
Easy data manipulation.
Extensive security models.
Supports Transactions and Full ACID guarantees; Atomic, Consistent, Isolated and Durable
Strong mathematical operations.
Different indexing capabilities.
Data modeling is schema based; each record in a relational database conforms to a schema – with a fixed number of fields (columns).

Those features are really good and indispensable if you are going to develop some enterprise software system, but yet these features have major impact on the performance.

There are cases where some of the above features can be dispensed in order to improve the performance. For example:

You don’t need relational data model

When you want to work with graph structures; although graph structures can be modeled in RDBMS, but this has serious query performance implications for recursive structures like for instance file trees and network structures, e.g. social graphs

You don’t need a complex security model

When your DB is a backend for a web application, and there may be just one user (the application server) connected to your DB at production time

You don’t need fixed schema (fixed number of fields for each record)

When your application depends on more flexible data model (schemaless) this allows you to freely add new fields without having to define changes. This is especially needed in Content Management Systems, where the metadata associated with the content can change dramatically

You don’t need the normalizations and JOIN operations

When faster analysis/reporting on extremely large databases is required; although the RDBMS star schema is used for speed but it still has problems with large joins if there is a need to join multiple large fact tables. Star schemas will outperform when it comes to extremely large databases but it is expensive, maintenance heavy, and usually requires big iron

You don’t need data manipulation

When the data can’t be changed after inserting it, for example in the Audit Trails and Tracking Systems.

You don’t need Transactions

When record-level atomicity with row-level locking is enough, for example you need a centralized database to store servers and applications logs. logging operations don't require complex atomicity.

NoSQL (Non-Relational Databases) paid a lot of attention to the trade-offs between consistency, fault-tolerance and performance to serve geographically distributed systems, low-latency or highly available applications. NoSQL’s primary goal is to achieve best performance and horizontal scalability. It attains this by reducing transactional semantics and referential integrity.

Today we have many NoSQL products and the following are the most common options:

HBase:

Don’t guarantee full ACID transactions; it only supports row-level atomicity.
Don’t support relational data model and thus don’t support JOIN operations
Support Free schema
Best used: for search engines and analyzing log data.

Cassandra:

Don’t guarantee full ACID transactions; it only supports row-level atomicity.
Support coarse-grained security model
Don’t support relational data model and thus don’t support JOIN operations
Supports data partitioning.
Don’t provide ad-hoc queries. Complex queries should be pre-planned
Best used: when you write more than you read (user activity logs)

MongoDB:

Don’t guarantee full ACID transactions; it only supports row-level atomicity and writes isolation.
Support data partitioning
Supports consistency: you can choose what level of consistency you want depending on the value of the data (e.g. faster performance = fire and forget inserts to MongoDB, slower performance = wait till insert has been replicated to multiple nodes before returning)
Don’t support relational data model and thus don’t support JOIN operations
Supports Document-oriented storage with free schema
Supports dynamic queries as well as aggregate functions
Best used: If you need dynamic queries and data analysis. It is also useful in the logging systems as well as content management systems.

Neo4J

Guarantee full ACID transactions.
Uses a graph model for data representation
Supports ad-hoc queries with powerful query language
Supports Online backup, advanced monitoring and High Availability
Best Used: For searching routes in social relations, public transport links, road maps, or network topologies.

Relational databases are suitable for 99% of the real world applications. You can practically do anything with relational databases. But, there are limitations on speed and scaling when it comes to massive high availability data stores.

NoSQL can give up some features of the relational databases for speed.

This doesn't mean the RDBMS is dead and it doesn't mean to choose either an RDBMS or NoSQL. The solution is to use different data storage technologies based upon the way data is being used by individual applications (Polyglot Persistence).

The following diagram shows an enterprise system where different applications use different data storage:

References:

Hung thread detection in WebSphere Application Server

2013-03-10T15:31:00.001+03:00

Websphere application server provides hung thread detection policy to detect the hung threads by sending notifications in the log files

Websphereusergroup.org | Joseph | Hung thread detection in WebSphere Application Server

StAXON - JSON via StAX

2013-03-10T15:18:00.001+03:00

StAXON project (APL2) which tries to give you the best from both worlds: JSON outside, but XML inside. One benefit from this is that you can integrate JSON with powerful XML-related technologies (Xpath, JAXB, and etc.) for free

StAXON - JSON via StAX | Javalobby

Websphere Application Server v7 - Extended Features

2013-03-10T14:59:00.000+03:00

Asynchronous beans(This feature will be included in JEE v7 Specs): An asynchronous bean is a Java object or enterprise bean that can be executed asynchronously by a Java 2 Platform Enterprise Edition (J2EE) application, using the J2EE context of the asynchronous bean creator.

Work object threads are fully supported by the application server and have the same properties as other asynchronous beans.
http://gee.cs.oswego.edu/dl/concurrencyee-interest/JavaOne-EEConcurrency.pdf

Object pools: provide an effective means of improving application performance at run time, by supporting the reuse of multiple instances of objects.
Startup beans (This feature will be included in EJB v3.1 specs): allow business logic to run when an EJB application starts or stops.
Dynamic cache: Caching the output of servlets, commands, and JavaServer Pages (JSP) improves application performance

DistributedMap and DistributedObjectCache: Supports objects replication among cluster instances

Oracle Hierarchical Queries

2013-01-29T16:14:00.001+03:00

If a table contains hierarchical data, then you can select rows in a hierarchical order using the hierarchical query clause:

hierarchical_query_clause::=

Hierarchical Queries

Performance Tuning Process

2012-12-03T12:46:00.001+03:00

Monitoring Enterprise Java Application

2012-11-29T15:27:00.001+03:00

Monitor	Alert	Technique	Type
New database connection is acquisitioned Database connection is closed	When the number of active connections exceeds threshold	Datasource proxy, log4jdbc, JAMon	Counter
New queue connection is acquisitioned Queue connection is closed	When the number of active connections exceeds threshold	Queue Connection Factory proxy	Counter
SQL processing time	When the processing time exceeds threshold	Connection proxy, PreparedStatement Proxy, log4jdbc, JAMon, Perf4J	Timing
Web service execution time	When the processing time exceeds threshold	Servlet Filter, Perf4J	Timing
New Web session is created Web Session is closed	When the number of active web sessions exceeds threshold	HttpSessionListener	Counter
New web service execution is started Web service execution is completed	When the number of active web service calls exceeds threshold	Servlet Filter	Counter
New EJB method call is received EJB method execution completed	When the active number of EJB methods executions exceeds threshold	EJB interceptors	Counter
EJB method execution	When the execution time exceeds threshold	EJB interceptors, Perf4J	Timing
Components availability	When component is not available	Heartbeat	Boolean
DB availability	When DB is not reachable for more than X seconds	Dummy SQL statement, Perf4J	Timing
Queue availability	When queue browsing is not applicable	Queue Browsing	Boolean
Number of messages in a queue	When the number of messages in the queue exceeds threshold	Queue Browsing	Counter
Message age	When the message age exceeds threshold	Queue Browsing	Number
Heap memory	When the heap memory usage exceeds threshold	JMX object java.lang:type=Memory	Number
System Exceptions/Errors	When error is raised	Exception Handling	List
Intrusions	When intrusion is detected	IDS tool	Intrusion
Deadlock	When deadlock is detected	JMX object java.lang:type=Threading	List

Tools to Analyze the Performance of IBM WAS

2012-11-26T15:00:00.002+03:00

Apache JMeter: Stress/Performace Testing Tool

Jmeter-plugins: Include many plugins; one of them is the stepping thread group plugin.

jstack: This is required to generate thread dumps on Sun/Oracle JVM

jstackSeries: This is a batch that can be used to generate series of thread dump files.

jmap: This is required to generate heap dump on Sun/Oracle JVM

jmapSeries: this is a batch that can be used to generate series of heap dump files.

IBM Performance Tuning Toolkit: This is required to generate heap/thread dumps on IBM JVM
IBM Thread and Monitor Dump Analyzer for Java (TMDA): This is required to analyze thread dump files
IBM Database Connection Pool Analyzer for IBM Websphere Application Server: this is required to analyze trace.log files generated by IBM WAS
IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer: This is required to analyze heap dump files
IBM Pattern Modeling and Analysis Tool for Java Garbage Collection (PMAT): Analyze garbage collection behaviour
JConsole: This is a visual JMX client that shows the CPU, Thread and Memory Usage for an Application:

topthread: This is a JConsole plugin to show the top threads that uses the CPU.
hibernate-jconsole: This is a JConsole plugin to show the hibernate entities and queries with few statistics.

log4jdbc: is a JDBC driver that can log SQL and/or JDBC calls (and optionally SQL timing information)
IBM Websphere Application Server - PMI: PMI is the core monitoring infrastructure for WebSphere Application Server and WebSphere family products like Portal, Commerce, and so on. The performance data provided by WebSphere PMI helps to monitor and tune the application server performance
TCP Load Balancer In Java: is a simple to use load balancer written in Java, which can be used for load balancing and fail-over clustering of TCP based services like HTTP, FTP, XMPP etc

Analyzing IBM Websphere Application Server Performance Issues - Files and Tools

2012-11-06T15:50:00.004+03:00

Performance Issue	Required Files for Analysis	How to Generate File using Sun JDK	How to Generate File using IBM JDK	Analysis Tool
Synchronized Blocking	Thread Dump (javacore files or native_stdout.log)	jstack jstack >> threaddumps.log	Using IBM Performance Tuning Toolkit	IBM Thread and Monitor Dump Analyzer for Java (TMDA)
DeadLock	Thread Dump (javacore files or native_stdout.log)	jstack jstack >> threaddumps.log	Using IBM Performance Tuning Toolkit	IBM Thread and Monitor Dump Analyzer for Java (TMDA)
High CPU Usage	Thread Dump (javacore files or native_stdout.log)	jstack jstack >> threaddumps.log	Using IBM Performance Tuning Toolkit	JConsole plugin topthreads
Connection Leak	trace.log	Enable the tracing using IBM Performance Tuning Toolkit	Using IBM Performance Tuning Toolkit	IBM Database Connection Pool Analyzer for IBM Websphere Application Server
Memory Leak	Heap Dump (.phd)	jmap -dump:format=b,file=	Using IBM Performance Tuning Toolkit or using wsadmin $AdminControl	IBM Monitoring and Diagnostic Tools for Java™ - Memory Analyzer
	(Garbage Collection Verbose)native_stdout.log	Enable Verbose garbage collection on Application servers > serverName > Process definition > Java Virtual Machine	Enable Verbose garbage collection using Performance Tuning Toolkit	IBM Monitoring and Diagnostic Tools for Java™ - Garbage Collection and Memory Visualizer
	(Garbage Collection Verbose)native_stdout.log			IBM Pattern Modeling and Analysis Tool for Java Garbage Collector (PMAT)

IBM Support Assistant Agent/Agent Manager v4.1 Installation Files

2012-10-18T10:25:00.001+03:00

IBM Support Assistant Agent/Agent Manager v4.1 Installation Files
IBM Support Assistant 2012/10/18 01:24:46

Java Concurrency Cheat Sheet from DZone Refcardz - Free, professional tutorial guides for developers

2012-10-11T16:54:00.001+03:00

Java Concurrency Cheat Sheet from DZone Refcardz - Free, professional tutorial guides for developers

java.dzone.com/sites/all/files/SOA_Service_Design_Cheat_Sheet.pdf

2012-10-11T16:53:00.001+03:00

This simple cheat sheet contains all the key goals, principals and design patterns that you should be aware of when designing SOA services and contains helpful links to places where you can find more in-depth information on each topic

java.dzone.com/sites/all/files/SOA_Service_Design_Cheat_Sheet.pdf

IBM WAS - Web container custom properties - HTTPOnly flag

2012-09-18T12:43:00.001+03:00

WebSphere Application Server Version 7.0

Web container custom properties - HTTPOnly flag

Harpoon Technologies: How to avoid SQL injection in Hibernate (A Hiberna...

2012-07-02T12:57:00.001+03:00

Harpoon Technologies: How to avoid SQL injection in Hibernate (A Hiberna...: Somewhere along the line java developers came to believe that Hibernate protects you from SQL injection. I'm not sure where they came to be...

Bye, Bye, 5 * 60 * 1000 //Five Minutes, Bye, Bye - Java Code Geeks

2012-06-28T13:42:00.001+03:00

Bye, Bye, 5 * 60 * 1000 //Five Minutes, Bye, Bye - Java Code Geeks

Software Patterns Architectural Patterns vs Design Patterns
Architectural Patterns (Common Architectural solutions for Common Architectural problems) Architectural pattern expresses a fundamental structural organization schema for software systems. It provides a set of predefined subsystems, specifies their responsibilities, and includes rules and guidelines for organizing the relationships between them. Below is a list of examples:		Design Patterns (Common Design solutions for Common Design problems) Design Pattern provides a scheme for refining the subsystems or components of a software system, or the relationships between them. It describes a commonly recurring structure of communicating components that’s solves a general design problem within a particular context. Below is a list of examples:
Structured Application Architecture	* MVC * Multi-Layered	OO Design Patterns (Can be applied within structured or distributed applications components/services built using OOP)	Examples: Decorator Builder * Observer * Façade * Command
Distributed Application Architecture	* N-Tier * SOA * Microservices
Integration Architecture	* Hub and Spoke * Pipes and Filters	Integration Design Patterns	Find Enterprise Integration Patterns blog for more details
		Design Patterns specific for particular Architectural Patterns	For each sub-type of the architectural patterns, there are a set of related design patterns: Enterprise Application Architecture Design Patterns, SOA Design Patterns, Microservices Design Patterns, etc.
		Concurrency Design Patterns (Can be applied to any Architecture)	Examples: * Thread Pool * Scheduler * Lock
		Security Design Patterns (Can be applied to any Architecture)	Examples * Authenticator * Input Validation

Haitham Raik

Software Architecture Evolution

Consolidated Design Rules and Standards for Great REST API

Consolidated Design Rules and Standards for Great Web API

OAuth and OpenID Connect Clarifications

OAuth in nutshell

OpenID Connect in nutshell

SAML at a high-level

Summary

API Definition and API History

Introduction

Definition

API Generations

Conclusion

Microservices Design Patterns (Arcitura vs Microservices.io vs Microsoft)

Enterprise Integration Patterns (Table)

Enterprise Integration Patterns

Integration Style

Root Pattern

Base Pattern

IIB Implementation Reference

File Transfer

Shared Database

Remote Procedure Call (RPC)

Messaging

Message

Command Message

Command Message Pattern in IIB

Document Message

Event Message

Request-Reply

Return Address

Correlation Identifier

Message Sequence

Message Expiration

Format Indicator

Message Channel (Async)

Point-to-Point

Publish-Subscribe

Datatype Channel

Invalid Message Channel

Dead Letter

Guaranteed Delivery

Channel Adapter

Message Bridge

Message Bus

Pipes and Filters

Message Router

Content-Based Router

Message Filter

Dynamic-Router

Recipient List

Splitter

Aggregator

Re-sequencer

Composed Message Processor

Scatter-Gather

Routing Slip

Process Manager

Message Broker

Message Translator

Envelope Wrapper

Content Enricher

Content Filter

Claim Check

Normalizer

Canonical Data Model

Message Endpoint

Messaging Gateway

Messaging Mapper

Transactional Client

Polling Consumer

Event-Driven Consumer

Competing Consumers

Message Dispatcher

Selective Consumer

Durable Subscriber

Idempotent Receiver

Service Activator

Desktop MVC vs Web MVC