Kiwi News critical database corruption writeup
AI summary of the linked articleThe writeup describes a Kiwi News node crash that left its Merkle trie in an inconsistent state, with the error "Missing node in DB" when reading the root e49c93db1dd3c722e8b3c75b07f2a5b61a01699f3e384437c7643772449129c4. The author found that trie.checkRoot returned false for that root, then loaded the prior root ddaae02d59d184e0f8278315499f5115c16d508bfe54c034f45e732cef0eb9fa, which the logs showed as the last good root. After calling trie.persistRoot(), checkRoot returned true, and the author suspects the program crashed partway through persisting the ethereumjs trie. |
Recommended by 1 curator |
1 comment  timSubmitter Hey, if you've used the site between 1 am and 2 am today (2024-02-01), then you might have noticed that it was down for an hour. This was due to a known corruption issue in the database that we don't exactly know where it comes from. We're still investigating. It took us a while to find a solution because corrupted databases are always really scary, and mistakes can be consequential. But by rolling back the trie's root to the prior root, we managed to recover all messages, and the site is now back up. This is very good because today was an all-time high in DAU, and it would have been so bad if that data on the achievement was now lost! |