Event Store logo

sss https://eventstore.org Menu

Developer Blog

Testing Event Store HA

  |   Written by: Greg Young   |   Articles

As we prepare to release Event Store 2.0.0 binaries (the source code was just merged to the master branch over on GitHub), we also will be opening up access to one of the test environments that we’ve been using for our commercial high availability clustered product, Event Store HA.

We get many questions asking how the clustered version behaves during network partitions and server failures – now you can see for yourself!

The cluster in the picture above is now connected to the web to allow people to see a cluster in action, as well as some of the testing we perform on them. This particular cluster is running a power outage test (and has been for some time). We can also test network partitions both of which the cluster can easily handle.

We pull the power on a node programmatically using the device seen here:

Cutting the power as opposed to just killing the process is a very important test for any database, as it tests the true durability of the data (including all the liars like Intel 520 SSDs, or sneaky operating system caching!). It also helps ensure our cluster election logic works reliably and quickly.

The test brings up a running cluster, then pulls the power on a random node in that cluster. Within about 60 seconds the node will start rebooting as power returns (the nodes themselves wait for power to be sustained to prevent damage due to cycling power). The node will then start up and verify its data before rejoining the cluster. After about one second of a node rejoining the test will rinse and repeat.

As of now, this particular cluster (Event Store HA 2.0.0 running on Windows and .NET) does roughly 1,000 power pulls per day.

Would we be comfortable pulling power in production? Absolutely!


Subscribe to the Event Store blog

Get the latest news and tutorials when they are released

You might also like

    Guest post: Subscription service for Event Store

      |   Written by: Steven Blair   |   Articles

    VME Retail have released version 1.0.0 of their Subscription Service, which is intended to make life that little bit easier for delivering events from Event Store to clients. The Subscription Service has been used privately by VME Retail Ltd for a few years in various solutions, and they have decided to open the code up to the rest of the Event Store community.

    Read article

    Case Study: How Event Sourcing is Spurring Linedata’s Digital Transformation

      |   Written by: Dave Remy   |   Articles

    Event sourcing is relatively new to the financial services industry. By implementing Event Store’s approach to support its digital transformation strategy, Linedata can be more responsive to the changing needs of its customers and gain a competitive edge in the market. “The core benefits of event-driven architecture allowed us to improve quality, performance and respond quickly to our customers,” said Adrian Tovey, who leads Linedata’s global program management. “Event Store has helped us meet the demand for change in the industry, a change in the way applications are used to deliver services to our customers; Linedata is at the forefront of those changes.”

    Read article

    Disabling disk caching in Ubuntu

      |   Written by: James Nugent   |   Articles

    Amongst the many interesting discussions I had at Build Stuff last week was about how it’s desirable to switch off disk caching for the disks used for Event Store databases to help ensure that data is durable in the face of power failures. This is actually true of many databases, indeed, postgres gives you a warning about the possible dire consequences of having write caching switched on when you may experience power failure. You can...

    Read article