Course Introduction

What You'll Learn in This Course

Golang Useful Resources and Tools

The authors

The history of Go

Installing Go

Golang: An Overview of Fundamental Concepts

Introduction

A Quick Primer on Key-Value Servers and Hash Tables

How to Create a Basic hello world Webserver with Go

Build a Basic hello world JSON API Server with Go + Chi

How to Create an In-Memory Hash Table API with Go + Chi

Conclusion

A key-value server

Testing a key-value server

What is End-to-End Testing, Integration Testing, and Unit Testing?

Writing Unit Tests for JSON Functions in Go

Parallel Testing in Golang: How to Use t.Parallel()

A Brief Overview of Golang Benchmarks

Integration testing

Further testing types

Distributed key-value server

What is Distributed Consensus and How Does it Work?

An Introduction to Using Raft for Distributed Consensus

Create a Distributed Webserver using Golang and Raft

Connecting Golang Servers With a Handler and Middleware

Building a Store Package to Sync Data in Go with Raft's FSM

Testing Distributed Golang Webservers in Docker Containers

Distributed Key Value Server

Monitoring

White Box Monitoring vs Black Box Monitoring

A Introduction to Golang Logging

Defining and Implementing Distributed System Tracing with Go

Alerting

Understanding the Basics of Canary Deployments

Wait, what's a load balancer?

What aren't we doing?

So what will we be doing?

How to Create a Basic Random-per-request Load Balancer

What is Weighted Round Robin Load Balancing?

Streamlining Configuration Updates for Canary Deployments

Other considerations

Three Open-Source Load Balancers Written in Go

A canary deployment server

Chat server

Chatting about chat servers

What features don't we want (yet)?

What features do we want?

Using a SQLite database in Golang

How to Create an API for a Golang Chat Server

Creating a Frontend Interface in Go for a Chat Server

How to Deploy a Golang Webserver With Docker

Future improvements

Future thinking

Project ideas

Areas of more research

In this book we will show you the some of the concerns of backend web development through the lens of Go.

It is taught by Nat Welch (formerly an SRE at Google) and Steve McCarthy (Etsy), two Go developers who have been writing software for more than 25 years combined.

In the course, we'll be covering the creation and concerns of a number of fundamental backend building blocks or larger applications:

- A key value server, including the use of distributed consensus algorithms
- A canary deployment server
- A real time chat service
- Testing and monitoring

Regardless of your background, Go can make you a _better backend developer_, but learning Go is challenging because the language looks simple on the outside, but you won't get its full benefit unless you know what it can _really_ do. 

**By the end of the first chapter, you'll already _have built a key-value server reminiscent of similar systems like Redis and memcached._** The subsequent chapters will dive into more hands-on projects and Go concepts including:

- How to test your Go systems with unit tests and integration tests
- **Project**: create a **distributed version of the key-value server** in order to solve the typical problems of server systems such as _node failure_
- How to solve the **problem of "consensus"** in distributed Go systems
- How to write code to **keep data in sync** when running a system for servers located around the globe
- Implementing **black box and white box monitoring** systems for Go servers
- **Project**: building a **load balancer server** from scratch
- Why and how to implement a **canary deployment server** for bug-free gradual deployments
- How to build a **configuration server** to allow zero-downtime deploys 
- Understanding the various tradeoffs in the **design decisions** for Go server systems
- **Project**: build a Go chat server system from the ground up 
- **Database modeling** for backend development as part of your chat server project
- **and a lot more**...

## A Key-Value Server Like Redis And Memcached

n the early chapters of the book we learn how to build a Golang version of a key-value server that works in a manner similar to Redis and Memcached. You'll implement each of these features in Go:

- set and retrieve values by key
- adding persistent storage
- using data encodings such as `JSON` and `base64`
- testing the server with unit tests and integration tests

In these chapters, we also learn how to build a more advanced solution for the problem: **a distributed version of the key-value server**. You will learn how to use **consensus algorithms** in Go to solve the problem of keeping distributed servers in sync. The solid grounding you get in programming Go in these first chapters readies you for more complex Go projects.

## A Real-Time Chat Service

A real-time chat server implemented in Go gives you the opportunity to bring together systems thinking as well as concerns common for web apps:

- relational data modeling with SQL
- authentication
- implementing RESTful resources

Working through this comprehensive project will give you **a high degree of confidence about using Go to implement a huge project from scratch**. You will feel ready to use Go with relational databases such as PostgreSQL or MySQL, and databases in general.

## A Canary Deployment Server And Load Balancer

The work you do in the chapters covering building a canary deployment server and load balancer will open your eyes to the full power of Go. Gradual deployments are a commonly used strategy for code deployments, and the projects you build in these chapters get you familiar with building this kind of **infrastructure software**. 

Your load balancer will implement _random-per-request_ as well as _round-robin_ load balancing. These projects will give you a thorough immersion in systems problem solving with Go. You'll be able to show off your skills on bold projects of your own, in job interviews, and on the job when collaborating with colleagues.

## This book is the _fastest_ way to learn modern backend Go development

With _The newline Guide to Reliable Webservers with Go_, you won't waste time reading shallow blog posts that can't go beyond showing you the simplest Go examples or Todo apps. Instead, **within the first chapter** you'll be well on your way to creating a **Redis-like key-value server**, and the complexity of the systems you're able to build only accelerates from there.

This book opens the door to **more effective backend development** and utilizing the full power of Golang for **microservices**. As Go transforms the systems development landscape, you can use this book and the skills it teaches you to contribute to a faster, more effective systems development experience for your team.

With _The newline Guide to Reliable Webservers with Go_ you can **go from novice to proficiency in Go in a predictable way, without wasting time, and without picking up bad Go coding habits**.

So grab a copy now to advance your journey with Go.

In this book we show important concepts in backend web development through the lens of Go.

Reliable Webservers with Go

The importance of testing your web servers

How to apply distributed consensus algorithms to your key value server

Let's write some unit tests as a way of getting familiar with Go's built-in testing library. Looking in 

, we can find the first function that isn't 

 and write a test for it.
So let us write a test for the 

 function, starting with making a failing test and running it.
We start by creating a 

.
Go will automatically know this is a test file because of the ending 

 lets us know that we'll be writing tests for 

package main

import "testing"

func TestJSON(t *testing.T) {
	t.Fatal("not implemented")
}

In the same directory as the test file we run our new test with 

 and we will see our expected failure from the 

$ go test
--- FAIL: TestJSON (0.00s)
    server_test.go:6: not implemented
FAIL
exit status 1
FAIL    github.com/fullstackio/reliable-go/kv-store     0.009s

Great! Now in order to replace that with something which will actually test the function, let's think about what the function is doing. It encodes arbitrary data and then writes it to a 

, that seems like a good type to test. So - let's test the message "hello world" in a map form. Add these lines at the top of the 

in := map[string]string{"hello": "world"}
out := `{"hello":"world"}`

 for the output we expect to see from it.

This is great and should be a good test, but we have an issue: 

 takes an input but it doesn't return an output. So we write to the response writer, but have no way of checking that data.
Luckily, Go's standard library has something that will help us: the 

 interface but also allows you to check what changes were made to it afterwards. If we use one of these, we can pass it with our message into 

recorder := httptest.NewRecorder()

JSON(recorder, in)

, giving us the ability to check those values.
For now let us only test that the body written to the recorder is the same as our input message. We can access it with its 

From that we can read the entire body, convert it to a string, and compare it with our input.

response := recorder.Result()
defer response.Body.Close()

got, err := io.ReadAll(response.Body)
if err != nil {
	t.Fatalf("Error reading response body: %s", err)
}

if string(got) != out {
	t.Errorf("Got %s, expected %s", string(got), out)
}

 here if we fail to read the body to indicate that our test has failed to run and we shouldn't continue. We always expect to be able to read it here, so if we can not, something is seriously wrong.

 to show that it has failed, but the other tests should continue.

package main

import (
	"io"
	"net/http/httptest"
	"testing"
)

func TestJSON(t *testing.T) {
	in := map[string]string{"hello": "world"}
	out := `{"hello":"world"}`
	recorder := httptest.NewRecorder()

	JSON(recorder, in)

	response := recorder.Result()
	defer response.Body.Close()

	got, err := io.ReadAll(response.Body)
	if err != nil {
		t.Fatalf("Error reading response body: %s", err)
	}

	if string(got) != out {
		t.Errorf("Got %s, expected %s", string(got), out)
	}
}

Our test is currently fairly simple. It only checks the body of the response, not any headers, and it only tests one message and one data type. Luckily, Go has a common pattern we can use to scale this up: 

Instead of having only one message to test, we can make a slice of them, with both our input and expected output. We can then use a 

 loop to run all of our tests one after the other. Adding a new case will then be as simple as adding another element to the slice - we'll do that shortly by adding the 

Before we do that, we need to convert our first test case into a table of test cases, including both our input and our expected output. Our input is a map of string to string, and our output is a string. So we need to create a slice of these inputs and outputs. Fortunately, we can use an anonymous struct, which is a struct that is defined without a name. It will let us create our desired slice and immediately create values to fill it. Also, because it is defined only in this one test case, future test cases can use their own anonymous structs if they have different inputs or outputs. The syntax looks slightly weird, because we are used to having our struct definition separate from instantiation. Combined with the fact that this is also a struct means there are a whole lot of curly braces. But it gives us exactly what we're looking for, which is great.

testCases := []struct {
	in  map[string]string
	out string
}{
	{map[string]string{"hello": "world"}, `{"hello":"world"}`},
}

In this section of code we are not defining a type for our struct but defining the struct inline. It's not necessarily the best practice, but so long as we are only using this structure here it should be fine. If we find ourselves wanting to use this in more than one test case we can convert it to a regular struct and reuse it.

 slice. We also create a loop around the main section of our test to loop through it with the one case we have provided. The loop encompasses everything from creating the recorder to the last 

for _, test := range testCases {
	recorder := httptest.NewRecorder()

	JSON(recorder, test.in)

	response := recorder.Result()
	defer response.Body.Close()

	got, err := io.ReadAll(response.Body)
	if err != nil {
		t.Fatalf("Error reading response body: %s", err)
	}

	if string(got) != test.out {
		t.Errorf("Got %s, expected %s", string(got), test.out)
	}
}

Now that we've converted our first test into a table-driven test, let us make sure it still works! 

We have one test passing, and now it will be very easy to add a second test case. Simply add the 

testCases := []struct {
	in  map[string]string
	out string
}{
	{map[string]string{"hello": "world"}, `{"hello":"world"}`},
	{map[string]string{"hello": "tables"}, `{"hello":"tables"}`},
}

---
title: "Writing Unit Tests for JSON Functions in Go"
# description: desc
# privateVideoUrl: url
code: https://gitlab.com/fullstackio/books/newline-course-apps/nat-welch-and-steve-mcarthy-reliable-webservers-in-go-app/-/tree/main/code/key-value-store/lesson_03.02
---

# A unit test for the JSON function

Let's write some unit tests as a way of getting familiar with Go's built-in testing library. Looking in `server.go`, we can find the first function that isn't `main` and write a test for it.
So let us write a test for the `JSON` function, starting with making a failing test and running it.
We start by creating a `server_test.go` file in the same directory as `server.go`.
Go will automatically know this is a test file because of the ending `_test.go`, and naming it starting with `server` lets us know that we'll be writing tests for `server.go` in it.

```go
package main

import "testing"

func TestJSON(t *testing.T) {
	t.Fatal("not implemented")
}
```

In the same directory as the test file we run our new test with `go test` and we will see our expected failure from the `t.Fatal`:

```bash
$ go test
--- FAIL: TestJSON (0.00s)
    server_test.go:6: not implemented
FAIL
exit status 1
FAIL    github.com/fullstackio/reliable-go/kv-store     0.009s
```

Great! Now in order to replace that with something which will actually test the function, let's think about what the function is doing. It encodes arbitrary data and then writes it to a `http.ResponseWriter`. So if we want to test it, we need to

-   supply the data
-   know what its encoded form looks like
-   check that it was written to the `http.ResponseWriter`

Because we are currently only using `JSON` to encode data of the type `map[string]string`, that seems like a good type to test. So - let's test the message "hello world" in a map form. Add these lines at the top of the `TestJSON` function:

```go
in := map[string]string{"hello": "world"}
out := `{"hello":"world"}`
```

I'm using the variable names `in` for the input we will pass to `JSON`, and `out` for the output we expect to see from it.

This is great and should be a good test, but we have an issue: `JSON` takes an input but it doesn't return an output. So we write to the response writer, but have no way of checking that data.
Luckily, Go's standard library has something that will help us: the `httptest` library - and it is [lovely!](https://golang.org/pkg/net/http/httptest/#ResponseRecorder)

`httptest.ResponseRecorder` is a struct that implements the `http.ResponseWriter` interface but also allows you to check what changes were made to it afterwards. If we use one of these, we can pass it with our message into `JSON` and then check the results after.

Use it and call `JSON` with it like this:

```go
recorder := httptest.NewRecorder()

JSON(recorder, in)
```

`JSON` will now have written to our `ResponseRecorder`, giving us the ability to check those values.
For now let us only test that the body written to the recorder is the same as our input message. We can access it with its `Result()` method which gives us the `http.Response` we normally get when making a request.

From that we can read the entire body, convert it to a string, and compare it with our input.

```go
response := recorder.Result()
defer response.Body.Close()

got, err := io.ReadAll(response.Body)
if err != nil {
	t.Fatalf("Error reading response body: %s", err)
}

if string(got) != out {
	t.Errorf("Got %s, expected %s", string(got), out)
}
```

Note that we use `t.Fatalf` here if we fail to read the body to indicate that our test has failed to run and we shouldn't continue. We always expect to be able to read it here, so if we can not, something is seriously wrong.

We use `t.Errorf` for our _actual test case_ to show that it has failed, but the other tests should continue.

Putting it all together looks like this:

```go
package main

import (
	"io"
	"net/http/httptest"
	"testing"
)

func TestJSON(t *testing.T) {
	in := map[string]string{"hello": "world"}
	out := `{"hello":"world"}`
	recorder := httptest.NewRecorder()

	JSON(recorder, in)

	response := recorder.Result()
	defer response.Body.Close()

	got, err := io.ReadAll(response.Body)
	if err != nil {
		t.Fatalf("Error reading response body: %s", err)
	}

	if string(got) != out {
		t.Errorf("Got %s, expected %s", string(got), out)
	}
}
```

### Enhancing the JSON unit test with tables

Our test is currently fairly simple. It only checks the body of the response, not any headers, and it only tests one message and one data type. Luckily, Go has a common pattern we can use to scale this up: **table-driven tests**!

Instead of having only one message to test, we can make a slice of them, with both our input and expected output. We can then use a `for` loop to run all of our tests one after the other. Adding a new case will then be as simple as adding another element to the slice - we'll do that shortly by adding the `{"hello":"tables"}` message.

Before we do that, we need to convert our first test case into a table of test cases, including both our input and our expected output. Our input is a map of string to string, and our output is a string. So we need to create a slice of these inputs and outputs. Fortunately, we can use an anonymous struct, which is a struct that is defined without a name. It will let us create our desired slice and immediately create values to fill it. Also, because it is defined only in this one test case, future test cases can use their own anonymous structs if they have different inputs or outputs. The syntax looks slightly weird, because we are used to having our struct definition separate from instantiation. Combined with the fact that this is also a struct means there are a whole lot of curly braces. But it gives us exactly what we're looking for, which is great.

```go
testCases := []struct {
	in  map[string]string
	out string
}{
	{map[string]string{"hello": "world"}, `{"hello":"world"}`},
}
```

In this section of code we are not defining a type for our struct but defining the struct inline. It's not necessarily the best practice, but so long as we are only using this structure here it should be fine. If we find ourselves wanting to use this in more than one test case we can convert it to a regular struct and reuse it.

With this struct, we can replace our `in` and `out` variables with `test.in` and `test.out`, which is our range variable over our `testCases` slice. We also create a loop around the main section of our test to loop through it with the one case we have provided. The loop encompasses everything from creating the recorder to the last `if` statement checking our output.

```go
for _, test := range testCases {
	recorder := httptest.NewRecorder()

	JSON(recorder, test.in)

	response := recorder.Result()
	defer response.Body.Close()

	got, err := io.ReadAll(response.Body)
	if err != nil {
		t.Fatalf("Error reading response body: %s", err)
	}

	if string(got) != test.out {
		t.Errorf("Got %s, expected %s", string(got), test.out)
	}
}
```