Study Topic: Decentralised MQ

From Matt Morris Wiki
Jump to navigation Jump to search

This is a Study Leave topic.

What Is It?

Message Queues allow asynchronous communication between software components.

The decentralised architecture puts both discovery and message persistence into separate components that are not centralised.

Why Study It?

ZeroMQ are the main player in the decentralised MQ space.

I'm going to be playing around with lots of languages and it will be potentially useful to have the intermediation handled by a proper MQ technology.

Toggl code

Toggl code is TECH-ZEROMQ

Deliverables

(DONE)

Max 7 hours of time:

  • install
  • "Hello World"
  • write up sections 1-5

Writeup

ØMQ - The Guide

Summary

0MQ works by connecting intelligent parts together. You can build AMQP style brokers out of it, but there is a great degree of extra flexibility.

If one was building some broker / messaging solutions then this would be an excellent approach to use.

The lowest level is various types of "smart sockets", which differ from TCP as follows:

  • they carry messages rather than a byte stream
  • IO is backgrounded
  • one-to-N routing behaviour is built in (according to the socket type)

0MQ offers unicast transports (inproc, ipc, tcp) and multicast (epgm, pgm) but best to stick to unicast unless you really know you need the multicase case!

By default 0MQ frames messages, although as of 3.3 there is a ZMQ_ROUTER_RAW option that allows non-framed data to be written.

Socket types

  • PUB - publish messages (send buffer), raw version XPUB
  • SUB - subscribe to messages (receive buffer), raw version XSUB
  • PUSH - add tasks to queue (send buffer)
  • PULL - get tasks from a queue (receive buffer)
  • REQ - make requests, expect replies (receive buffer)
  • REP - expect requests, send replies (receive buffer)
  • ROUTER - accept incoming messages (send+receive buffers)
  • DEALER - distribute outgoing messages (send+receive buffers)
  • PAIR - pass signals between threads (send+receive buffers)

And here are what go together:

  • PUB and SUB
  • XPUB, XSUB: raw versions of PUB and SUB
  • PUSH and PULL: add and pull tasks to/from a queue
  • PAIR and PAIR
  • REQ (can go with REP and ROUTER)
  • REP (can go with REQ and DEALER)
  • ROUTER (can go with REQ, DEALER and ROUTER)
  • DEALER (can go with REP, DEALER and ROUTER)

There are then convenience functions at higher leves:

  • zmq_proxy() to put ROUTER/DEALER, XSUB/XPUB, PULL/PUSH together

Use send_multipart(), recv_multipart() to get pub/sub envelopes

Buffers work with High-Water Marks that default to 1,000 but can also be set

More on Routers:

  • ROUTER sockets will drop messages silently by default if they don't know where to send them
  • Can use ZMQ_ROUTER_MANDATORY to force an error on no-can-send (EHOSTUNREACH)

Envelopes

See the start of section 3 of the guide for more on envelopes:

  • Frames are byte-length prefixed
  • A req/reply envelope is zero of more reply addresses, followed by an empty frame, followed by zero or more message body frames
  • So the simplest is an empty delimiter frame + a single message frame
  • The crucial difference between socket types is how they deal with frames:
    • REQ sends (and expects) an empty delimiter frame. Synchronous: one peer at a time, even if multiple are connected
    • REP saves the identity frames, passes message to caller, restores frame on response. Synchronous: one peer at a time, even if multiple are connected.
    • DEALER: oblivious to envelopes. Distribute sent to all connections, fair-queue received from all connections
    • ROUTER: when receiving, creates connection ID and passes on as first frame, when sending uses the first message part to get the address

Some natural flows

  • Extended request-reply: [REQ] - ROUTER - DEALER - [REP]
  • Message broker: [Client:REQ] - ROUTER - ROUTER - [REQ:Worker]
  • Async client-server: [Client:DEALER] - ROUTER:Server:DEALER - [DEALER:Worker]
  • Can federate brokers: A broker can have local set with client & then both cloud + state setup with broker

Reliable Request-Reply Patterns

Named after pirates because of the "RRR" acronym

Chapter 4 opens with a discussion of "reliability", making the point that something can only be "reliable" with regard to explicitly stated failure modes, the main ones of which are (in descending order of probability):

  • Application code (crashes, freezes, slowdoens, memory exhaustion)
  • System code (as application code, but should be more reliable)
  • Message queue overflow (leading to lost messages)
  • Network connection failure (again, leading to lost messages)
  • Hardware failure (taking down all processes on the box)
  • Exotic network failure (ports on a switch, etc)
  • Physical disasters (e.g. natural disasters taking out entire data centres)

Patterns:

  • Lazy Pirate Protocol (single server: client can handle server restarts)
  • Simple Pirate Protocol (multiple workers: client can handle worker restarts, central router is point of failure)
  • Paranoid Pirate Protocol ("PPP": multiple workers: heartbeating so can handle queue restart)

Heartbeating discussion: various options are covered around pp156-159. Some points:

  • Build it in at the start; retro-fitting is difficult
  • Trace message flow while developing
  • Suitable timing might range from 10ms to 30s depending on requirements
  • Use the lowest (quickest) interval if there is a range across peers - do not use an infinite timeout
  • Heartbeat on the message socket so the heartbeats act as a keep-alive: helps stop firewalls killing silent connections

Protocols: once things get more complicated, you should write the contracts out explicitly. A wiki is at http://rfc.zeromq.org and you can see an example spec at http://rfc.zeromq.org/spec:6

More Patterns:

  • Majordomo Protocol ("MDP": named services)
  • Asynchronous Majordomo Protocol (higher throughput)
  • MMI: Majordomo management interface for service discovery (http://rfc.zeromq.org/spec:8)
  • TitanicProtocol : layered on MDP, adding persistence
  • Binary Star Protocol (high-availability pair)
    • "Binary Star Reactor" packages the idea into a reusable "bstar" API
  • Freelance Protocol (multiple clients, multiple servers, no central broker)
    • Various connection options are explored: summarised as "simple, brutal, complex or nasty" :)

Pub-Sub Patterns

Recepients should not talk back to senders - crucual for scalability.

Concepts need to align with the standard Pragmatic General Multicast (PGM) network multicast protocol.

  • Espresso Protocol (tracing pub-sub)
  • Last Value Caching (LVC) proxy
  • Suicidal Snail Protocol (slow subscribers kill themselves) - other alternatives are
    • Queue on publisher - but then publisher is vulnerable
    • Queue on subscriber - far better but doesn't help if subscriber is just too slow in general
    • Stop queueing after a while - but then messages are dropped
    • Disconnect subscribers - but that's hard to layer on top if subscribers don't talk to publishers (see above)
  • Black Box Protocol (high-speed subscribers)
  • Clone Protocol (central server, reliable unless it goes down)
    • eventually adds Binary Star protocol for dealing with server failover as well
  • Clustered Hashmap Protocol (distributed hashmap across clients)

Also discussed

  • Subtrees (e.g. a/b/c/d topics)
  • Ephemeral values (these expiry automatically unless refreshed)
Retrieved from "http://www.matthew-morris.com/wiki/index.php?title=Study_Topic:_Decentralised_MQ&oldid=166299"