Review of ‘AWS re:Invent 2015 | (ARC309) Microservices: Evolving Architecture Patterns in the Cloud’ presentation.
From monolithic to microservices, on AWS.
Ruby on rails -> Java based functional services.
Java based functional services resulted in requirement for everything to be loaded into memory – 20 minutes to start services. Still very large services – with many engineers working on it. This means that commits to those repos take a long time to QA. So a commit is made then start working on something else then a week or two later have to fix what you hardly remember. Then to get specific information out of those big java apps it was required to parse the entire homepage. So…
Big sql database is still there – This will mean changes with schema change are difficult to do without outages. Included in this stage of micro services was:
- Team autonomy – give teams a problem and they can build whatever services they need
- Voluntary adoption – tools/techniques/processes
- Goal driven initiatives
- Failing fast and openly
What are the services? – Email, shipping cost, recommendations, admin, search
Anatomy of a service:
A service has its own datastore and it completely owns its own datastore. This is where dependency on one big schema is avoided. Services at gilt.com average 2000 lines of code and 32 source files.
Service discovery – enormously simple? – Discovery is a client needs to get to a services, how is it going to get there. ‘It has the name of the service – look up that URL’.
Use ZooKeeper as a highly available store.
Moving all this to the cloud via ‘lift and shift’ with AWS Direct Connect. In AWS all services where their own ec2 instance and dockerized.
Being a good citizen in a microservices organisation:
- Service Consumer
- Design for failure
- Expect to be throttled
- Retry with exponential backoff
- Degrade gracefully
- Cache when appropriate
- Service Provider
- Publish your metrics
- Throughput, error rate, latency
- Protect yourself (throttling)
- Implementation details private
- Maintain backwards compatability
- See Amazon API gateway
- Publish your metrics
Again service discovery – simple with naming conventions, DNS and load balancers. Avoid DNS issues with dynamic service registry (ZooKeeper, Eureka, Consul, SmartStack).
Data management – moving away from the schema change problems. And the other problems (custom stored procedures, being stuck with one supplier, single point of failure). Microservices must include decentralisation of datastores, services own their datastores and do not share them. This has a number of benefit from being able to chose whatever data store technology best meets the service needs, make changes without affecting other services, scale the data stores independently. So how do we ensure transactional integrity? Distributed locking sounds horrible. Do all services really need strict transactional integrity? Use queues to retry later.
Aggregation of data – I need to do analytics on my data? AWS Kenesis firehose, Amazon SQS, custom feed.
Continuous Delivery/Deployment – Introduced some AWS services Code Deploy — or just use Jenkins.
How many services per container/instance – Problems with monitoring granularity, scaling is less granular, ownership is less atomic, continuos deployment is tougher with immutable containers.
I/O Explosion – Mapping dependencies between services is tough. Some will be popular/hotspots. Service consumers need to cache were they can. Dependency injection is also an option – You can only make a request from services A if you have the required data from service B and C in your request.
Monitoring – Logs are key, also tracing request through fanned dependency can be much easier with a requirement for a header that is passed on.
Unique failures – Watched a day in the life of a Netflix engineers… good points on failures. We accept and increased failure rate to maintain velocity. We just want to ensure failures are unique. For this to happen we need to have open and safe feedback.