Course Introduction

What You'll Learn in This Course

Golang Useful Resources and Tools

The authors

The history of Go

Installing Go

Golang: An Overview of Fundamental Concepts

Introduction

A Quick Primer on Key-Value Servers and Hash Tables

How to Create a Basic hello world Webserver with Go

Build a Basic hello world JSON API Server with Go + Chi

How to Create an In-Memory Hash Table API with Go + Chi

Conclusion

A key-value server

Testing a key-value server

What is End-to-End Testing, Integration Testing, and Unit Testing?

Writing Unit Tests for JSON Functions in Go

Parallel Testing in Golang: How to Use t.Parallel()

A Brief Overview of Golang Benchmarks

Integration testing

Further testing types

Distributed key-value server

What is Distributed Consensus and How Does it Work?

An Introduction to Using Raft for Distributed Consensus

Create a Distributed Webserver using Golang and Raft

Connecting Golang Servers With a Handler and Middleware

Building a Store Package to Sync Data in Go with Raft's FSM

Testing Distributed Golang Webservers in Docker Containers

Distributed Key Value Server

Monitoring

White Box Monitoring vs Black Box Monitoring

A Introduction to Golang Logging

Defining and Implementing Distributed System Tracing with Go

Alerting

Understanding the Basics of Canary Deployments

Wait, what's a load balancer?

What aren't we doing?

So what will we be doing?

How to Create a Basic Random-per-request Load Balancer

What is Weighted Round Robin Load Balancing?

Streamlining Configuration Updates for Canary Deployments

Other considerations

Three Open-Source Load Balancers Written in Go

A canary deployment server

Chat server

Chatting about chat servers

What features don't we want (yet)?

What features do we want?

Using a SQLite database in Golang

How to Create an API for a Golang Chat Server

Creating a Frontend Interface in Go for a Chat Server

How to Deploy a Golang Webserver With Docker

Future improvements

Future thinking

Project ideas

Areas of more research

In this book we will show you the some of the concerns of backend web development through the lens of Go.

It is taught by Nat Welch (formerly an SRE at Google) and Steve McCarthy (Etsy), two Go developers who have been writing software for more than 25 years combined.

In the course, we'll be covering the creation and concerns of a number of fundamental backend building blocks or larger applications:

- A key value server, including the use of distributed consensus algorithms
- A canary deployment server
- A real time chat service
- Testing and monitoring

Regardless of your background, Go can make you a _better backend developer_, but learning Go is challenging because the language looks simple on the outside, but you won't get its full benefit unless you know what it can _really_ do. 

**By the end of the first chapter, you'll already _have built a key-value server reminiscent of similar systems like Redis and memcached._** The subsequent chapters will dive into more hands-on projects and Go concepts including:

- How to test your Go systems with unit tests and integration tests
- **Project**: create a **distributed version of the key-value server** in order to solve the typical problems of server systems such as _node failure_
- How to solve the **problem of "consensus"** in distributed Go systems
- How to write code to **keep data in sync** when running a system for servers located around the globe
- Implementing **black box and white box monitoring** systems for Go servers
- **Project**: building a **load balancer server** from scratch
- Why and how to implement a **canary deployment server** for bug-free gradual deployments
- How to build a **configuration server** to allow zero-downtime deploys 
- Understanding the various tradeoffs in the **design decisions** for Go server systems
- **Project**: build a Go chat server system from the ground up 
- **Database modeling** for backend development as part of your chat server project
- **and a lot more**...

## A Key-Value Server Like Redis And Memcached

n the early chapters of the book we learn how to build a Golang version of a key-value server that works in a manner similar to Redis and Memcached. You'll implement each of these features in Go:

- set and retrieve values by key
- adding persistent storage
- using data encodings such as `JSON` and `base64`
- testing the server with unit tests and integration tests

In these chapters, we also learn how to build a more advanced solution for the problem: **a distributed version of the key-value server**. You will learn how to use **consensus algorithms** in Go to solve the problem of keeping distributed servers in sync. The solid grounding you get in programming Go in these first chapters readies you for more complex Go projects.

## A Real-Time Chat Service

A real-time chat server implemented in Go gives you the opportunity to bring together systems thinking as well as concerns common for web apps:

- relational data modeling with SQL
- authentication
- implementing RESTful resources

Working through this comprehensive project will give you **a high degree of confidence about using Go to implement a huge project from scratch**. You will feel ready to use Go with relational databases such as PostgreSQL or MySQL, and databases in general.

## A Canary Deployment Server And Load Balancer

The work you do in the chapters covering building a canary deployment server and load balancer will open your eyes to the full power of Go. Gradual deployments are a commonly used strategy for code deployments, and the projects you build in these chapters get you familiar with building this kind of **infrastructure software**. 

Your load balancer will implement _random-per-request_ as well as _round-robin_ load balancing. These projects will give you a thorough immersion in systems problem solving with Go. You'll be able to show off your skills on bold projects of your own, in job interviews, and on the job when collaborating with colleagues.

## This book is the _fastest_ way to learn modern backend Go development

With _The newline Guide to Reliable Webservers with Go_, you won't waste time reading shallow blog posts that can't go beyond showing you the simplest Go examples or Todo apps. Instead, **within the first chapter** you'll be well on your way to creating a **Redis-like key-value server**, and the complexity of the systems you're able to build only accelerates from there.

This book opens the door to **more effective backend development** and utilizing the full power of Golang for **microservices**. As Go transforms the systems development landscape, you can use this book and the skills it teaches you to contribute to a faster, more effective systems development experience for your team.

With _The newline Guide to Reliable Webservers with Go_ you can **go from novice to proficiency in Go in a predictable way, without wasting time, and without picking up bad Go coding habits**.

So grab a copy now to advance your journey with Go.

In this book we show important concepts in backend web development through the lens of Go.

Reliable Webservers with Go

The importance of testing your web servers

How to apply distributed consensus algorithms to your key value server

Our API is going to be relatively simple. We need to serve our HTML pages, handle login and the creation of users, and send and retrieve messages.

The first part of the API is ensuring that we have authenticated users.

We have the ability to check a user's password to a hashed and salted version saved in the database. However, we don't want to have to do that on every request, because that would require the user to send their password to us every time they took any action. Doing that would either be incredibly cumbersome for the user, or very insecure, as we would have to store their password somewhere to send it on every request.

Instead, we will do what everyone else does, and set up a 

 for the user using cookies. We have a couple options here.

 for each user. This essentially gives the user a cookie with a unique ID that we can use to look up their information every time they make a request. This is a very common and sensible way to handle user authentication.

The only drawback is that we have to store that information somewhere, which centralizes the looking up of user information. For example we could create a 

 struct and store it in our database the same way we store users and messages. Then we would simply do a lookup every time a user makes a request.

Or we could go with a second option, using a 

 that includes user information in it. The primary benefit of generating this kind of cookie rather than a session token is that when a user makes a request, we don't need to look up their session information in the database or some other data store. We only need to verify that their signed token is valid, and then we know that the information it contains can be trusted, and we can use that.

For session tokens, you always have to look up the session information from a data store. Even if that data store is designed specifically for that kind of lookup, like the key-value store we wrote, it still involves a network request, and creates an additional point of failure.

Signed cookies containing data, on the other hand, don't require the additional request nor even the storage of session information. That may seem like a huge win until you consider that all the information in the session has to be stored somewhere, and that somewhere is in the cookie, meaning every request a user makes sends all of that data, regardless of how much or little of it you need. So although there is a somewhat significant advantage, it's not free.

In order to simplify our setup and not have to store and retrieve sessions, we are going to go with the second option, cryptographically-signed cookies. Specifically, we are going to use 

s. Going into exactly how JWTs work is beyond the scope of this book, but if you want to find out more you should check out 

JWTs will require the use of a library to provide the necessary pieces for creating, validating and pulling information from tokens. Luckily, a library has been written to make it easier to use JWTs with 

The main thing the library needs to work is a struct called 

. Conveniently, all the functions we will use are defined from this struct, and we can use a pointer to it so we can use it in a number of places. This means for us, it's exactly like our database connection, so we should put it in the same place in 

type Config struct {
	DB        *gorm.DB
	TokenAuth *jwtauth.JWTAuth
}

Now that we have that essential piece of information, let's go back to 

The only thing we need to store in the token is the user's name. Since we won't be looking up any user information, we don't need to store their ID or any other information. Making the token will involve calling 

func (c *Config) MakeToken(name string) string {
	_, tokenString, _ := c.TokenAuth.Encode(map[string]interface{}{"user_name": name})
	return tokenString
}

Because the token is JSON, it accepts any kind of information in a map so long as there are strings for keys. In order to be descriptive, we will store this information as 

, but we could have used any string for it.

The only remaining piece left to do is retrieve this user name from the token. For this, we actually won't need 

 middleware function that will store all the necessary information in the 

. Unsurprisingly, it also provides a method to retrieve the token data directly from context. Using that, we can write a function to get the user name from the context.

func GetUserNameFromContext(ctx context.Context) string {
	_, c, _ := jwtauth.FromContext(ctx)
	return c["user_name"].(string)
}

 method returns us the token itself (which we don't actually care about), the token data, and an error in case there was an issue retrieving the data. We are explicitly ignoring the error here, because we will already have validated the token by the time we call this function, so we can be sure it won't fail if it gets this far.

 functions done, we are ready to move on to the server itself.

We have all of the surrounding pieces ready, so now we can make our server and all of the handlers we're going to need.

The first step is to decide what routes we need our server to have, so let's enumerate them and their paths.

So in all there will be six routes. We also have to consider which of these need to be behind authentication and which don't. Luckily that is a pretty simple decision - users will need to be logged in so they can see the chat as well as send and receive messages. The other routes will not. So from the list above, our first three routes don't require authentication, and the last three do require authentication.

package main

import (
	// We will have various standard library imports here eventually

	"github.com/go-chi/chi"
	"github.com/go-chi/chi/middleware"
	"github.com/go-chi/jwtauth"
)

func main() {
	// We will fill this in later
}

func (c *Config) SetupRoutes() *chi.Mux {
	r := chi.NewRouter()
	r.Use(middleware.Logger)
	r.Get("/", IndexHandler)
	r.Post("/newuser", c.NewUserPostHandler)
	r.Post("/login", c.LoginPostHandler)
	r.Route("/", func(r chi.Router) {
		r.Use(jwtauth.Verifier(c.TokenAuth))
		r.Use(jwtauth.Authenticator)

		r.Get("/chat", APIChatHandler)
		r.Route("/api", func(r chi.Router) {
			r.Get("/messages", c.APIMessagesHandler)
			r.Post("/messages", c.APIMessagesPostHandler)
		})
	})
	return r
}

Most of this should look similar to what we have done before, but there are a couple of new things:

 so we have access to both our JWT library and our database. It also lets us pass that same information into all of our handlers that need the database without having to use a global variable or create a database connection in the handler.

 provides that allows us to group methods that require similar functionality or path prefixes. We are using it twice here, once to set up authentication for all of our routes that need it, and a second time to group both of our messages endpoints, which we are putting under the 

 along with either message route to have authentication, and the message routes' paths will both be 

---
title: "How to Create an API for a Golang Chat Server"
# description: desc
# privateVideoUrl: url
---

# Our API

Our API is going to be relatively simple. We need to serve our HTML pages, handle login and the creation of users, and send and retrieve messages.

The first part of the API is ensuring that we have authenticated users.

#### Let's talk about authentication

We have the ability to check a user's password to a hashed and salted version saved in the database. However, we don't want to have to do that on every request, because that would require the user to send their password to us every time they took any action. Doing that would either be incredibly cumbersome for the user, or very insecure, as we would have to store their password somewhere to send it on every request.

Instead, we will do what everyone else does, and set up a _session_ for the user using cookies. We have a couple options here.

The first option is to set up **session cookies** for each user. This essentially gives the user a cookie with a unique ID that we can use to look up their information every time they make a request. This is a very common and sensible way to handle user authentication.

The only drawback is that we have to store that information somewhere, which centralizes the looking up of user information. For example we could create a `Session` struct and store it in our database the same way we store users and messages. Then we would simply do a lookup every time a user makes a request.

Or we could go with a second option, using a **cryptographically-signed cookie** that includes user information in it. The primary benefit of generating this kind of cookie rather than a session token is that when a user makes a request, we don't need to look up their session information in the database or some other data store. We only need to verify that their signed token is valid, and then we know that the information it contains can be trusted, and we can use that.

Neither of these options is perfect.

For session tokens, you always have to look up the session information from a data store. Even if that data store is designed specifically for that kind of lookup, like the key-value store we wrote, it still involves a network request, and creates an additional point of failure.

Signed cookies containing data, on the other hand, don't require the additional request nor even the storage of session information. That may seem like a huge win until you consider that all the information in the session has to be stored somewhere, and that somewhere is in the cookie, meaning every request a user makes sends all of that data, regardless of how much or little of it you need. So although there is a somewhat significant advantage, it's not free.

In order to simplify our setup and not have to store and retrieve sessions, we are going to go with the second option, cryptographically-signed cookies. Specifically, we are going to use **JSON Web Tokens**, or **JWT**s. Going into exactly how JWTs work is beyond the scope of this book, but if you want to find out more you should check out [jwt.io](https://jwt.io).

#### Adding JWTs

JWTs will require the use of a library to provide the necessary pieces for creating, validating and pulling information from tokens. Luckily, a library has been written to make it easier to use JWTs with `chi`: `github.com/go-chi/jwtauth`.

The main thing the library needs to work is a struct called `JWTAuth`. Conveniently, all the functions we will use are defined from this struct, and we can use a pointer to it so we can use it in a number of places. This means for us, it's exactly like our database connection, so we should put it in the same place in `database.go`.

```go
type Config struct {
	DB        *gorm.DB
	TokenAuth *jwtauth.JWTAuth
}
```

Now that we have that essential piece of information, let's go back to `crypto.go` and define our methods for _making the token_ and _retrieving information_ from it.

The only thing we need to store in the token is the user's name. Since we won't be looking up any user information, we don't need to store their ID or any other information. Making the token will involve calling `Encode` on our `JWTAuth` that we've stored as `TokenAuth` in our `Config`.

```go
func (c *Config) MakeToken(name string) string {
	_, tokenString, _ := c.TokenAuth.Encode(map[string]interface{}{"user_name": name})
	return tokenString
}
```

Because the token is JSON, it accepts any kind of information in a map so long as there are strings for keys. In order to be descriptive, we will store this information as `"user_name"`, but we could have used any string for it.

The only remaining piece left to do is retrieve this user name from the token. For this, we actually won't need `TokenAuth` directly, because the `jwtauth` library provides a `chi` middleware function that will store all the necessary information in the `context.Context` which is included in the `http.Request`. Unsurprisingly, it also provides a method to retrieve the token data directly from context. Using that, we can write a function to get the user name from the context.

```go
func GetUserNameFromContext(ctx context.Context) string {
	_, c, _ := jwtauth.FromContext(ctx)
	return c["user_name"].(string)
}
```

The `jwtauth.FromContext` method returns us the token itself (which we don't actually care about), the token data, and an error in case there was an issue retrieving the data. We are explicitly ignoring the error here, because we will already have validated the token by the time we call this function, so we can be sure it won't fail if it gets this far.

With those final `crypto.go` functions done, we are ready to move on to the server itself.

### The server routes

We have all of the surrounding pieces ready, so now we can make our server and all of the handlers we're going to need.

The first step is to decide what routes we need our server to have, so let's enumerate them and their paths.

-   GET `/` - the login page
-   POST `/login` - the login handler
-   POST `/newuser` - the new user creation handler
-   GET `/chat` - the chat page
-   GET `/api/messages` - the get messages route
-   POST `/api/messages` - the send messages route, differentiated with the above by using POST requests

So in all there will be six routes. We also have to consider which of these need to be behind authentication and which don't. Luckily that is a pretty simple decision - users will need to be logged in so they can see the chat as well as send and receive messages. The other routes will not. So from the list above, our first three routes don't require authentication, and the last three do require authentication.

In a new file, `server.go`, we can start to set everything up.

```go
package main

import (
	// We will have various standard library imports here eventually

	"github.com/go-chi/chi"
	"github.com/go-chi/chi/middleware"
	"github.com/go-chi/jwtauth"
)

func main() {
	// We will fill this in later
}

func (c *Config) SetupRoutes() *chi.Mux {
	r := chi.NewRouter()
	r.Use(middleware.Logger)
	r.Get("/", IndexHandler)
	r.Post("/newuser", c.NewUserPostHandler)
	r.Post("/login", c.LoginPostHandler)
	r.Route("/", func(r chi.Router) {
		r.Use(jwtauth.Verifier(c.TokenAuth))
		r.Use(jwtauth.Authenticator)

		r.Get("/chat", APIChatHandler)
		r.Route("/api", func(r chi.Router) {
			r.Get("/messages", c.APIMessagesHandler)
			r.Post("/messages", c.APIMessagesPostHandler)
		})
	})
	return r
}
```

Most of this should look similar to what we have done before, but there are a couple of new things:

First, we're calling `SetupRoutes` on `Config` so we have access to both our JWT library and our database. It also lets us pass that same information into all of our handlers that need the database without having to use a global variable or create a database connection in the handler.

The other new piece is the `r.Route`. This is a helper method `chi` provides that allows us to group methods that require similar functionality or path prefixes. We are using it twice here, once to set up authentication for all of our routes that need it, and a second time to group both of our messages endpoints, which we are putting under the `/api` path prefix.

So this sets up `/chat` along with either message route to have authentication, and the message routes' paths will both be `/api/messages`.

### Filling in the routes

We have just defined our list of routes - now it is time to fill them in.

Let's start with our IndexHandler, which is going to be incredibly simple. All it needs to do is serve our index page.

```go
func IndexHandler(w http.ResponseWriter, r *http.Request) {
	http.ServeFile(w, r, "index.html")
}
```

Just to make things easy, we are going to put our `index.html` file in the same place as the rest of the files, because there aren't that many of them. But if this were a larger application, we would probably put it in a directory specifically for HTML or potentially templates.

The index page itself will also be rather simple. We only need it for two things, creating a user and logging in, so we can have two forms, one pointing to `LoginPostHandler` and the other to `NewUserPostHandler`. Sticking with the common web practice of having a user confirm their password when creating an account, the primary difference between the two forms will be a "Confirm Password" field for user creation.

```html
<!DOCTYPE html>
<html lang="en">
  <head> </head>
  <body>
    <div style="font-size: 1.2em;">Login</div>
    <form method="post" action="/login">
      <div>
        <label for="user">Name</label>
        <input type="text" name="user" />
      </div>
      <div>
        <label for="password">Password</label>
        <input type="password" name="password" />
      </div>
      <div>
        <button type="submit">Login</button>
      </div>
    </form>

    <div style="font-size: 1.2em;">Register</div>
    <form method="post" action="/newuser">
      <div>
        <label for="user">Name</label>
        <input type="text" name="user" />
      </div>
      <div>
        <label for="password">Password</label>
        <input type="password" name="password" />
      </div>
      <div>
        <label for="password">Confirm Password</label>
        <input type="password" name="password_confirm" />
      </div>
      <div>
        <button type="submit">Create User</button>
      </div>
    </form>
  </body>
</html>
```