Study Writeup: WEB APIs

This is a Study Leave write-up.

What Is It?

Extension out from REST to consider web APIs more generally:

• designing the APIs themselves
• the architecture behind them so they function scalable
• relationship between internal and external consumption

Why Study It?

Fundamental part of architecture - what do the points of connection look like, and how should they best be enabled?

Toggl code

Toggl code is WRITEUP-WEB-API

Deliverables

Max 3 hours of time. Deliver write-up on this page: first draft now done.

Write-Up

See here for REST specifically: Study Writeup: REST APIs

Business Models

• Free
  • A tiny faction of business models
• Developer Pays
  • Pay as you go (e.g. AWS EC2)
  • Tiered (lower cost per unit as volumes increase)
  • Freemium
  • Unit-based (different features are priced seperately)
  • Transaction fee (e.g. Paypal, Stripe)
• Developer Is Paid
  • Revenue share: dev get a share of referred purchases
  • Affiliate (e.g. Amazon)
  • Sign-up/referral (can be one-time or recurring)
• Indirect
  • Facilitate acquisition of valuable content (Twitter/eBay)
  • SaaS
  • Internal value

API Design

Some general points

• There's a lot of debate over contracts: people don't want the SOAP/WSDL strongly-typed contract, but some kind of contract is necessary, even in hypermedia style returns. Question is how to get contracts that allow systems to evolve.
• Try to build in affordance: guidance on usage, rather than forcing clients to figure out and code up logic.
• Think about distance in time (how long will this be used for?) as well as space (how can people who never meet me understand this?)
• API management should concentrate on lifecycle management - understanding usage and enabling deprecation and change.
• API governance needs to keep things loosely coupled, rather than being overly prescriptive in governing the details of operations and functor names.
• Try to avoid versioning: hypermedia is key here
• Focus on the message rather than the client or sender

Compliance: where there's an audit type compliance regime, we need

• lifecycle control so only permissible versions are in production
• repeatability so everyone concerned knows what versions are currently in play
• adequate authorisation and authentication

Examples

Netflix

Netflix started off with classic RESTful architecture, which had some problems

• A lowest-common-denominator solution, not great for all clients
• Slowed innovation as all endpoints where used by everyone
• Slowed innovation because API team became a bottleneck
• Too much network chat and suboptimal payload efficiency
• Had to start putting in complex feature flags to cater for client differences

In the new structure

• Endpoints are dynamically defined at runtime and new ones can be dev+tested+canaried_deployed by each team
• Common RESTful Endpoint Management API handling publishing endpoint code to Casssandra multi-region cluster
• JWM runtime so can use e.g. Groovy and many others
• Async API
• Functional Reactive Programming Model (allows quantities that vary in response to events to be captured in an FP framework)
• Hysterix fault-tolerance layer
• API Service Layer thinks about functional contexts rather than specific systems
• "Circuit Breaker" to automatically trip fallback if requests time out, pools/queues hit 100%, or there's a client exception
• "Chaos Monkey": randomly kill production machines
• "Latency Monkey": inject failures and high latency into dependent service calls

Cloud considerations:

• Three copies of everything, so you can lose an entire zone and you're OK
• Allow greatly varying speeds of deployment between teams: a new service might deploy 10 times in a day!
• Try to avoid big changes on Fridays though :)
• Deployment:
  • Canary pattern/smoke test: start spinning with one
  • tell Amazon auto-scaler to copy 500 times (takes 8 minutes!)
  • switch over traffic, keeping old machines around
  • get rid of old machines once have something like 8 hours of stability on new set
• Their enterprise-scale PAS is huge compared to what suits most shops (Heroku, Cloud Foundry)
• Launched new service (Edda) that describes complete infrastructure, run every minute, gives versioned history of state
• Need garbage collector on Amazon entities etc
• Very JWM based in language choices: using Scala more at the moment
• Lots of Github/open sourcing
• Lots of cloud monitoring and management tools!

Twitter

• Continually monitor production
• Continually stress test produciton
• Have "runbooks" for every system so know what to do in emergency
• Central "Decider" to turn things on and off without needing to reply (or can do partial deploy, say 50% of users)
• Everyone deploys at different rates: could be daily, monthly
• Finegrained SOA
• Very similar "canary system" then ramp-up to what Netflix do
• HDFS, MapReduce, Hadoop on data in long-term storage

REST API Design and Documentation Tools

Swagger is a framework that allows definition of REST API functionality via JSON docs:

• interactive API docs
• client-side code generation in multiple languages
• server-side code generation in Java and Scala

This can be used to generate docs around an existing API: however one can start by writing the Swagger spec and use the products to help with an interface-driven development.

Also there's API Blueprint and RAML (RESTful API Modelling Language)

See here for a comparison (July2014 RAML vs. Swagger vs. API Blueprint

OAuth and its discontents

OAuth Wikipedia page

• Nov 2006: OAuth starts at Twitter
• Apr 2007: Discussion group created
• Jul 2007: Initial spec drafted
• Dec 2007: OAuth 1.0 final draft released
• Apr 2009: OAuth 1.0a released to correct security hole
• Apr 2010: OAuth 1.0 protocol published as RFC5849
• Jul 2012: Eran Hammer quits
  • The lead author and editor of the OAuth 2.0 network authorisation standard used by Facebook, Twitter and Google has downed tools and walked out, taking his name off the specification in the process
  • "At the end, I reached the conclusion that OAuth 2.0 is a bad protocol,” Hammer writes. “WS-* bad. It is bad enough that I no longer want to be associated with it."
• Oct 2012: OAuth 2.0 main framework published as RFC6749
  • Because OAuth 2.0 is more of a framework than a defined protocol, OAuth 2.0 implementations are less likely to be naturally interoperable with any other OAuth 2.0 implementations.

UDDI and its discontents

Universal Description, Discovery and Integration (UDDI, pronounced /ˈjʊdiː/) is a platform-independent, Extensible Markup Language (XML)-based registry by which businesses worldwide can list themselves on the Internet, and a mechanism to register and locate web service applications

UDDI was written in August 2000, at a time when the authors had a vision of a world in which consumers of web services would be linked up with providers through a public or private dynamic brokerage system.

UDDI was included in the Web Services Interoperability (WS-I) standard as a central pillar of web services infrastructure, and the UDDI specifications supported a publicly accessible Universal Business Registry in which a naming system was built around the UDDI-driven service broker.

UDDI has not been as widely adopted as its designers had hoped:

• IBM, Microsoft, and SAP announced they were closing their public UDDI nodes in January 2006.
• The group defining UDDI, the OASIS Universal Description, Discovery, and Integration (UDDI) Specification Technical Committee voted to complete its work in late 2007 and has been closed
• In September 2010, Microsoft announced they were removing UDDI services from future versions of the Windows Server operating system. Instead, this capability would be moved to Biztalk.

UDDI systems are most commonly found inside companies, where they are used to dynamically bind client systems to implementations, However, much of the search metadata permitted in UDDI is not used for this relatively simple role.

Hysterix

Hysterix is a library that helps control the interactions between distributed services by adding latency tolerance and fault tolerance logic. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve the system’s overall resiliency.

Hystrix is designed to do the following:

• Give protection from and control over latency and failure from dependencies accessed (typically over the network) via third-party client libraries.
• Stop cascading failures in a complex distributed system.
• Fail fast and rapidly recover.
• Fallback and gracefully degrade when possible.
• Enable near real-time monitoring, alerting, and operational control.