This function checks if a given node should be marked as FAIL. * It happens if the following conditions are met: * * 1) We received enough failure reports from other master nodes via gossip. * Enough means that the majority of the masters signaled the node is * down recently. * 2) We believe this node is in PFAIL state. * * If a failure is detected we also inform the whole cluster ab
| 1320 | * FAIL flag will be cleared after some time. |
| 1321 | */ |
| 1322 | void markNodeAsFailingIfNeeded(clusterNode *node) { |
| 1323 | int failures; |
| 1324 | int needed_quorum = (g_pserver->cluster->size / 2) + 1; |
| 1325 | |
| 1326 | if (!nodeTimedOut(node)) return; /* We can reach it. */ |
| 1327 | if (nodeFailed(node)) return; /* Already FAILing. */ |
| 1328 | |
| 1329 | failures = clusterNodeFailureReportsCount(node); |
| 1330 | /* Also count myself as a voter if I'm a master. */ |
| 1331 | if (nodeIsMaster(myself)) failures++; |
| 1332 | if (failures < needed_quorum) return; /* No weak agreement from masters. */ |
| 1333 | |
| 1334 | serverLog(LL_NOTICE, |
| 1335 | "Marking node %.40s as failing (quorum reached).", node->name); |
| 1336 | |
| 1337 | /* Mark the node as failing. */ |
| 1338 | node->flags &= ~CLUSTER_NODE_PFAIL; |
| 1339 | node->flags |= CLUSTER_NODE_FAIL; |
| 1340 | node->fail_time = mstime(); |
| 1341 | |
| 1342 | /* Broadcast the failing node name to everybody, forcing all the other |
| 1343 | * reachable nodes to flag the node as FAIL. |
| 1344 | * We do that even if this node is a replica and not a master: anyway |
| 1345 | * the failing state is triggered collecting failure reports from masters, |
| 1346 | * so here the replica is only helping propagating this status. */ |
| 1347 | clusterSendFail(node->name); |
| 1348 | clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE|CLUSTER_TODO_SAVE_CONFIG); |
| 1349 | } |
| 1350 | |
| 1351 | /* This function is called only if a node is marked as FAIL, but we are able |
| 1352 | * to reach it again. It checks if there are the conditions to undo the FAIL |
no test coverage detected