Build fault-tolerant Calculator with OTP

Elixir/Erlang inbuilt provides us OTP (Open Telecom Platform) which is a set of libraries that ships with Erlang. OTP is not that much about telecom anymore (It's more about software that has the property of telecom applications, but yeah.)

Elixir/Erlang developers use OTP to build robust, fault-tolerant applications.

Let's build a Calculator with GenServer behavior with OTP. Later we will also use Supervisor behavior to restart the killed process. We will also see how to use ets i.e. Erlang Term Storage, an inbuilt memory store that comes with Elixir/Erlang. We will use ets to maintain the state, when the process is restarted by a Supervisor.

Create a mix project for Calculator

$ mix new calci --sup
* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/calci.ex
* creating lib/calci/application.ex
* creating test
* creating test/test_helper.exs
* creating test/calci_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd calci
    mix test

Run "mix help" for more commands.

Run tests

$ mix test
Compiling 2 files (.ex)
Generated calci app
..

Finished in 0.02 seconds
1 doctest, 1 test, 0 failures

Randomized with seed 375143

Write Calculator Test

Create calculator_test.exs in test/calci folder

└── test
    ├── calci
    │   └── calculator_test.exs
    ├── calci_test.exs
    └── test_helper.exs

Make calculator_test.exs look as below

defmodule Calci.CalculatorTest do
  use ExUnit.Case
end

Let’s write our first test

defmodule Calci.CalculatorTest do
  use ExUnit.Case

  test "add 100" do
    calc = Calculator.new
    new_calc = Calculator.add(calc, 100)

    assert 100 == Calculator.value(new_calc)
  end
end

Run test, test should fail

$ mix test
Compiling 1 file (.ex)
Generated calci app
..

  1) test add 100 (Calci.CalculatorTest)
     test/calci/calculator_test.exs:12
     ** (UndefinedFunctionError) function Calci.Calculator.new/0 is undefined (module Calci.Calculator is not available)
     code: calc = Calculator.new()
     stacktrace:
       Calci.Calculator.new()
       test/calci/calculator_test.exs:13: (test)


Finished in 0.03 seconds
1 doctest, 2 tests, 1 failure

Randomized with seed 515834

Make test pass

Let’s write a Calculator module

Create a Calculator module at calci/lib/calci/calculator.ex

and add a new function to initialise calculator with 0

defmodule Calci.Calculator do
  alias Calci.Calculator

  defstruct value: 0

  def new do
    %Calculator{value: 0}
  end

  def add(%Calculator{value: old_value}, number) do
    %Calculator{value: old_value + number}
  end

  def get(%Calculator{value: value}) do
    value
  end
end

And here is test

defmodule Calci.CalculatorTest do
  use ExUnit.Case

  alias Calci.Calculator

  test "add 100" do
    calc = Calculator.new()
    new_calc = Calculator.add(calc, 100)

    assert 100 == Calculator.get(new_calc)
  end
end

Now when you run the test, it should pass

$ mix test
Compiling 1 file (.ex)
...

Finished in 0.03 seconds
1 doctest, 2 tests, 0 failures

Randomized with seed 576402

Introduce GenServer

Using GenServerhandle_call

Let’s introduce GenServer here to maintain state now.

defmodule Calci.Calculator do
  use GenServer

  alias Calci.Calculator

  defstruct value: 0

  ...

  def init(_initial_value) do
    {:ok, Calculator.new()}
  end

  def handle_call({:add, new_number}, _from, state) do
    {:reply, "add #{new_number}", Calculator.add(state, new_number)}
  end

  def handle_call(:get, _from, state) do
    :timer.sleep(1000)
    {:reply, Calculator.get(state), state}
  end
end

iex(1)> alias Calci.Calculator
Calci.Calculator

iex(2)> {:ok, pid} = GenServer.start_link(Calculator, 0, [])
{:ok, #PID<0.142.0>}

iex(3)> GenServer.call(pid, {:add, 400})
"add 400"
iex(4)> GenServer.call(pid, :get)
400

Add 400 again

iex(5)> GenServer.call(pid, {:add, 400})
"add 400"
iex(6)> GenServer.call(pid, :get)
800

This is working fine.

But when we run our test suite with

Tests are still failing

 $  mix test
# failing test

We should not pass state, but depend on GenServer to maintain our state.

Let’s change test for not passing state.

defmodule Calci.CalculatorTest do
  use ExUnit.Case

  alias Calci.Calculator

  test "add 100" do
    Calculator.new()
    Calculator.add(100)

    assert 100 == Calculator.get()
  end
end

 mix test
..

  1) test add 100 (Calci.CalculatorTest)
     test/calci/calculator_test.exs:12
     ** (UndefinedFunctionError) function Calci.Calculator.add/1 is undefined or private
     code: Calculator.add(100)
     stacktrace:
       (calci) Calci.Calculator.add(100)
       test/calci/calculator_test.exs:14: (test)



Finished in 0.04 seconds
1 doctest, 2 tests, 1 failure

Randomized with seed 588954

Naming the processes

iex(1)> alias Calci.Calculator
Calci.Calculator

iex(2)> {:ok, pid} = GenServer.start_link(Calculator, Calculator.new, [])
{:ok, #PID<0.143.0>}

iex(3)> GenServer.call(pid, {:add, 400})
"add 400"

iex(4)> Process.register pid, Calculator
true

iex(5)> Process.whereis(Calculator)
#PID<0.143.0>

iex(6)> GenServer.call(Calculator, {:add, 900})
"add 900"

iex(7)> GenServer.call(Calculator, {:add, 10})
"add 10"

iex(8)> GenServer.call(Calculator, :get)
1310

Instead of doing manually we can pass following option

name: __MODULE__ in GenServer.start_link to register current module name to refer to GenServer PID.

def start_link(args // []) do
  GenServer.start_link(__MODULE__, 0, name: __MODULE__)
end	

Let’s fix tests by adding changes in Calculator

defmodule Calci.Calculator do
  use GenServer

  alias Calci.Calculator

  defstruct value: 0

  # API
  def start_link(_arg \\ []) do
    GenServer.start_link(__MODULE__, Calculator.new(), name: __MODULE__)
  end

  def new do
    %Calculator{value: 0}
  end

  def add(number) do
    GenServer.call(Calculator, {:add, number})
  end

  def get() do
    GenServer.call(Calculator, :get)
  end

  # Callbacks

  def init(init_value) do
    {:ok, init_value}
  end

  def handle_call({:add, new_number}, _from, state) do
    {:reply, "add #{new_number}", _add(state, new_number)}
  end

  def handle_call(:get, _from, state) do
    :timer.sleep(1000)
    {:reply, _get(state), state}
  end

  # Private functions

  defp _add(%Calculator{value: old_value}, number) do
    %Calculator{value: old_value + number}
  end

  defp _get(%Calculator{value: value}) do
    value
  end
end

And here is how our CalculatorTest looks like

defmodule Calci.CalculatorTest do
  use ExUnit.Case

  alias Calci.Calculator

  # Runs before each test
  setup do
    Calculator.start_link()
    {:ok, %{}}
  end

  test "add 100" do
    Calculator.add(100)

    assert 100 == Calculator.get()
  end

  test "add 100 twice" do
    Calculator.add(100)
    Calculator.add(100)
    result = Calculator.get()

    assert 200 == result
  end
end

Now run the tests

$ mix test
....

Finished in 2.0 seconds
1 doctest, 3 tests, 0 failures

Randomized with seed 786247

Use of callbacks handle_cast

Let’s understand and make use of handle_cast callback available in GenServer behaviour

We can add a reset function in our Calculator which will allow us to reset the number in Calculator to 0

Let’s write a test

  test "reset to 0" do
    Calculator.add(100)
    Calculator.reset()

    result = Calculator.get()

    assert 0 == result
  end

defmodule Calci.Calculator do
  ...
  def reset() do
    GenServer.cast(Calculator, :reset)
  end
  ...
  def handle_cast(:reset, _state) do
    {:noreply, Calculator.new()}
  end
end

Remaining tests (subtract, multiply, divide)

Let’s add more tests and get Calculator working fine.

  test "subtract 100" do
    Calculator.add(100)
    Calculator.subtract(100)

    result = Calculator.get()

    assert 0 == result
  end

  test "multiple by 5" do
    Calculator.add(100)
    Calculator.multiply(5)

    result = Calculator.get()

    assert 500 == result
  end

  test "divide by 100" do
    Calculator.add(300)
    Calculator.divide(100)

    result = Calculator.get()

    assert 3 == result
  end

Our Calculator would have following new functions now

  def subtract(number) do
    GenServer.call(Calculator, {:subtract, number})
  end

  def multiply(number) do
    GenServer.call(Calculator, {:multiply, number})
  end

  def divide(number) do
    GenServer.call(Calculator, {:divide, number})
  end

...

  def handle_call({:subtract, number}, _from, state) do
    {:reply, "subtract #{number}", _subtract(state, number)}
  end

  def handle_call({:multiply, number}, _from, state) do
    {:reply, "multiply #{number}", _multiply(state, number)}
  end

  def handle_call({:divide, number}, _from, state) do
    {:reply, "divide #{number}", _divide(state, number)}
  end

...

  defp _subtract(%Calculator{value: old_value}, number) do
    %Calculator{value: old_value - number}
  end

  defp _multiply(%Calculator{value: old_value}, number) do
    %Calculator{value: old_value * number}
  end

  defp _divide(%Calculator{value: old_value}, number) do
    %Calculator{value: old_value / number}
  end

Now run the tests

$ mix test
........

Finished in 6.0 seconds
1 doctest, 7 tests, 0 failures

Randomized with seed 276060

Let it crash

Let’s try to divide by 0

iex(1)> alias Calci.Calculator
Calci.Calculator

iex(2)> Calculator.start_link([])
{:ok, #PID<0.142.0>}

iex(3)> Calculator.add(100)
"add 100"

iex(4)> Calculator.get
100

iex(5)> Process.whereis(Calculator) # To check process present
#PID<0.151.0>. 

iex(6)> Calculator.divide(0)

16:29:42.885 [error] GenServer Calci.Calculator terminating
** (ArithmeticError) bad argument in arithmetic expression
    (calci) lib/calci/calculator.ex:47: Calci.Calculator.handle_call/3
    (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.139.0>): {:divide, 0}
State: 100
Client #PID<0.139.0> is alive

    (stdlib) gen.erl:167: :gen.do_call/4
    (elixir) lib/gen_server.ex:1007: GenServer.call/3
    (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir) src/elixir.erl:275: :elixir.eval_forms/4
    (iex) lib/iex/evaluator.ex:257: IEx.Evaluator.handle_eval/5
    (iex) lib/iex/evaluator.ex:237: IEx.Evaluator.do_eval/3
    (iex) lib/iex/evaluator.ex:215: IEx.Evaluator.eval/3
    (iex) lib/iex/evaluator.ex:103: IEx.Evaluator.loop/1
** (EXIT from #PID<0.139.0>) shell process exited with reason: an exception was raised:
    ** (ArithmeticError) bad argument in arithmetic expression
        (calci) lib/calci/calculator.ex:47: Calci.Calculator.handle_call/3
        (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
        (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.9.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(7)> Calculator.get
** (UndefinedFunctionError) function Calculator.get/0 is undefined (module Calculator is not available)
    Calculator.get()

We can see that Calculator process is no longer available. That’s the issue.

iex(1)> Process.whereis(Calculator)
nil

Adding supervisor

In your mix project, you must have this file

calci/`lib/calci/application.ex

Just add {Calci.Calculator, []} in the children array

defmodule Calci.Application do
  use Application

  def start(_type, _args) do
    children = [
      {Calci.Calculator, []}
    ]

    opts = [strategy: :one_for_one, name: Calci.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Now when we try

iex -S mix

iex(1)> alias Calci.Calculator
Calci.Calculator

iex(2)> Calculator.start_link([])
{:error, {:already_started, #PID<0.139.0>}}
iex(3)>

That’s the difference.

iex(3)> Calculator.add(100)
"add 100"

iex(4)> Calculator.add(100)
"add 100"

iex(5)> Calculator.get
200

iex(6)> Calculator.divide(0)

16:34:19.663 [error] GenServer Calci.Calculator terminating
** (ArithmeticError) bad argument in arithmetic expression
    (calci) lib/calci/calculator.ex:47: Calci.Calculator.handle_call/3
    (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.140.0>): {:divide, 0}
State: 200
Client #PID<0.140.0> is alive

    (stdlib) gen.erl:167: :gen.do_call/4
    (elixir) lib/gen_server.ex:1007: GenServer.call/3
    (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir) src/elixir.erl:275: :elixir.eval_forms/4
    (iex) lib/iex/evaluator.ex:257: IEx.Evaluator.handle_eval/5
    (iex) lib/iex/evaluator.ex:237: IEx.Evaluator.do_eval/3
    (iex) lib/iex/evaluator.ex:215: IEx.Evaluator.eval/3
    (iex) lib/iex/evaluator.ex:103: IEx.Evaluator.loop/1
** (exit) exited in: GenServer.call(Calci.Calculator, {:divide, 0}, 5000)
    ** (EXIT) an exception was raised:
        ** (ArithmeticError) bad argument in arithmetic expression
            (calci) lib/calci/calculator.ex:47: Calci.Calculator.handle_call/3
            (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
            (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
            (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
    (elixir) lib/gen_server.ex:1010: GenServer.call/3

But now when you do

iex(6)> Calculator.get
0

Calculator PID is still alive, we can check

iex(7)> Process.alive?(Process.whereis(Calculator))
true

iex(2)> Calculator.add(100)
"add 100"
iex(3)> Calculator.add(100)
"add 100"

iex(4)> Calculator.get
200

That's good. We are now able to restart our Calculator process.

But after crash our Calculator state is resetting to 0.

We need to maintain state when process is restarted. For this Elixir (Erlang) offers an inbuilt memory store named ets (Erlang Term Storage)

Let's add ets support

Final version

As it's inbuilt in Elixir language, we don't need to add any library depdencency. WE can start with following steps.

Create ets table in application.ex

defmodule Calci.Application do
  use Application

  defp init_ets() do
    :ets.new(:calculator, [:set, :public, :named_table])
  end

  def start(_type, _args) do
    children = [
      {Calci.Calculator, []}
    ]
    opts = [strategy: :one_for_one, name: Calci.Supervisor]
    init_ets()
    Supervisor.start_link(children, opts)
  end
end

defmodule Calci.Calculator do
  use GenServer

  alias Calci.Calculator

  defstruct value: 0
  
  ...  
  def init(init_value) do
    lookup()
  end
  ...

  defp _add(%Calculator{value: old_value}, number) do
    result = old_value + number
    store(result)
    %Calculator{value: result}
  end

  ...

  defp store(result) do
    :ets.insert(:calculator, {:result, %Calculator{value: result}})
  end

  defp lookup() do
    case :ets.lookup(:calculator, :result) do
      [{:result, result}] -> {:ok, result}
      [] -> {:ok, %Calculator{value: 0}}
    end
  end
end

After adding ets support, when we run

iex(1)> Calculator.add(100)
Add 100
iex(2)> Calculator.add(100)
Add 100
iex(1)> Calculator.get
200
iex(5)> Calculator.divide(0)

Calculator process would crash again with exception of ArithmeticError

But we when we run, we should get our value in Calculator process back.

iex(1)> Calculator.get
200

This is because, before Calculator process was crashed, we had stored it's state in ets After the Calculator process is restarted, we can still get the value back as state is properly reinitialized for the new process.

That's how we restarted the process and recovered the state.