Interrupt block validation on new best head

[heifner]

Provide a mechanism to auto recover from "poison" blocks. Poison blocks are blocks with transactions which take an unexpectedly long time to execute. Poison block validation will automatically be interrupted when a better block is received from the network.

If a BP node, always produce when scheduled to do so. Interrupting any block validation in progress. This restores the old behavior of "drop late blocks". When a producer is scheduled to produce it will interrupt any validation in progress and produce on top of the previous block.

• For BP nodes, interrupt block validation when scheduled to produce
  • Moved producer_plugin timer off the main thread executor and onto its own thread.
• Interrupt block validation when a new best head* is received from the network.
  • Do not apply blocks from the fork database on startup. Blocks in the fork database have not been validated. If one of them would happen to be a poison block then a node would be stuck attempting to validate the block. By allowing network blocks to begin flowing into the node immediately on startup, a new best head can be received and interrupt block validation of blocks being processed out of the fork database.
• Ctrl-c/signal interrupt now leaves block in the fork database instead of purging it and its descendants.
  • The block and its descendants can be left in the fork database because a node now auto recovers when a new block is received from the network. Or is interrupted if a BP and is scheduled to produce.

• New best head means that a fork switch is required. If the new best head is a descendant of the current block being validated then validation is not interrupted. Only when the new block is on an alternative fork is validation be interrupted.

Resolves #1039
Resolves #1040

[ericpassmore]

Note:start
category: System Stability
component: Internal
summary: Interrupt block validation on new best head, enables auto recovery from poison blocks.
Note:end