Course Introduction

What You'll Learn in This Course

Golang Useful Resources and Tools

The authors

The history of Go

Installing Go

Golang: An Overview of Fundamental Concepts

Introduction

A Quick Primer on Key-Value Servers and Hash Tables

How to Create a Basic hello world Webserver with Go

Build a Basic hello world JSON API Server with Go + Chi

How to Create an In-Memory Hash Table API with Go + Chi

Conclusion

A key-value server

Testing a key-value server

What is End-to-End Testing, Integration Testing, and Unit Testing?

Writing Unit Tests for JSON Functions in Go

Parallel Testing in Golang: How to Use t.Parallel()

A Brief Overview of Golang Benchmarks

Integration testing

Further testing types

Distributed key-value server

What is Distributed Consensus and How Does it Work?

An Introduction to Using Raft for Distributed Consensus

Create a Distributed Webserver using Golang and Raft

Connecting Golang Servers With a Handler and Middleware

Building a Store Package to Sync Data in Go with Raft's FSM

Testing Distributed Golang Webservers in Docker Containers

Distributed Key Value Server

Monitoring

White Box Monitoring vs Black Box Monitoring

A Introduction to Golang Logging

Defining and Implementing Distributed System Tracing with Go

Alerting

Understanding the Basics of Canary Deployments

Wait, what's a load balancer?

What aren't we doing?

So what will we be doing?

How to Create a Basic Random-per-request Load Balancer

What is Weighted Round Robin Load Balancing?

Streamlining Configuration Updates for Canary Deployments

Other considerations

Three Open-Source Load Balancers Written in Go

A canary deployment server

Chat server

Chatting about chat servers

What features don't we want (yet)?

What features do we want?

Using a SQLite database in Golang

How to Create an API for a Golang Chat Server

Creating a Frontend Interface in Go for a Chat Server

How to Deploy a Golang Webserver With Docker

Future improvements

Future thinking

Project ideas

Areas of more research

In this book we will show you the some of the concerns of backend web development through the lens of Go.

It is taught by Nat Welch (formerly an SRE at Google) and Steve McCarthy (Etsy), two Go developers who have been writing software for more than 25 years combined.

In the course, we'll be covering the creation and concerns of a number of fundamental backend building blocks or larger applications:

- A key value server, including the use of distributed consensus algorithms
- A canary deployment server
- A real time chat service
- Testing and monitoring

Regardless of your background, Go can make you a _better backend developer_, but learning Go is challenging because the language looks simple on the outside, but you won't get its full benefit unless you know what it can _really_ do. 

**By the end of the first chapter, you'll already _have built a key-value server reminiscent of similar systems like Redis and memcached._** The subsequent chapters will dive into more hands-on projects and Go concepts including:

- How to test your Go systems with unit tests and integration tests
- **Project**: create a **distributed version of the key-value server** in order to solve the typical problems of server systems such as _node failure_
- How to solve the **problem of "consensus"** in distributed Go systems
- How to write code to **keep data in sync** when running a system for servers located around the globe
- Implementing **black box and white box monitoring** systems for Go servers
- **Project**: building a **load balancer server** from scratch
- Why and how to implement a **canary deployment server** for bug-free gradual deployments
- How to build a **configuration server** to allow zero-downtime deploys 
- Understanding the various tradeoffs in the **design decisions** for Go server systems
- **Project**: build a Go chat server system from the ground up 
- **Database modeling** for backend development as part of your chat server project
- **and a lot more**...

## A Key-Value Server Like Redis And Memcached

n the early chapters of the book we learn how to build a Golang version of a key-value server that works in a manner similar to Redis and Memcached. You'll implement each of these features in Go:

- set and retrieve values by key
- adding persistent storage
- using data encodings such as `JSON` and `base64`
- testing the server with unit tests and integration tests

In these chapters, we also learn how to build a more advanced solution for the problem: **a distributed version of the key-value server**. You will learn how to use **consensus algorithms** in Go to solve the problem of keeping distributed servers in sync. The solid grounding you get in programming Go in these first chapters readies you for more complex Go projects.

## A Real-Time Chat Service

A real-time chat server implemented in Go gives you the opportunity to bring together systems thinking as well as concerns common for web apps:

- relational data modeling with SQL
- authentication
- implementing RESTful resources

Working through this comprehensive project will give you **a high degree of confidence about using Go to implement a huge project from scratch**. You will feel ready to use Go with relational databases such as PostgreSQL or MySQL, and databases in general.

## A Canary Deployment Server And Load Balancer

The work you do in the chapters covering building a canary deployment server and load balancer will open your eyes to the full power of Go. Gradual deployments are a commonly used strategy for code deployments, and the projects you build in these chapters get you familiar with building this kind of **infrastructure software**. 

Your load balancer will implement _random-per-request_ as well as _round-robin_ load balancing. These projects will give you a thorough immersion in systems problem solving with Go. You'll be able to show off your skills on bold projects of your own, in job interviews, and on the job when collaborating with colleagues.

## This book is the _fastest_ way to learn modern backend Go development

With _The newline Guide to Reliable Webservers with Go_, you won't waste time reading shallow blog posts that can't go beyond showing you the simplest Go examples or Todo apps. Instead, **within the first chapter** you'll be well on your way to creating a **Redis-like key-value server**, and the complexity of the systems you're able to build only accelerates from there.

This book opens the door to **more effective backend development** and utilizing the full power of Golang for **microservices**. As Go transforms the systems development landscape, you can use this book and the skills it teaches you to contribute to a faster, more effective systems development experience for your team.

With _The newline Guide to Reliable Webservers with Go_ you can **go from novice to proficiency in Go in a predictable way, without wasting time, and without picking up bad Go coding habits**.

So grab a copy now to advance your journey with Go.

In this book we show important concepts in backend web development through the lens of Go.

Reliable Webservers with Go

The importance of testing your web servers

How to apply distributed consensus algorithms to your key value server

Observability is a popular term for the overarching work of making sure you can tell what is going on with your service. In the world of observability, there are two ways to observe your system: 

. Black box monitoring is the idea that you have no control over the system, and so you examine it from the outside. White box monitoring is the idea that you have full control of the system, and you can instrument internal pieces with metrics, logs and traces to understand how things are working.

As we said, black box monitoring is examining from the outside. It is often also called 

. We are going to generate a synthetic event, send it through our system and see what happens. This differs from the integration testing that we discussed in Chapter 3 because we are sending our synthetic event to our production environment, instead of a test environment. That being said, this is essentially testing, because we have a predefined synthetic input, and we will have a test that will validate the output.

Synthetic events for web systems tend to be HTTP requests, which means the easiest synthetic event generator in this case is the command line application 

. Curl has been around since 1997, and is an essential tool. The command flags I use most frequently are 

. Together you can watch how a request works.

So if your server is running locally on port 8080, you could call 

curl -svL localhost:8080 > /dev/null 2>&1 | grep HTTP

 every second and watch the responses come in. For reference, 

 will only show us output lines that include the string 

This would, of course, get boring. So instead you could add the 

 command, which runs commands every n seconds.

$ watch -d -n 1 "curl -svL localhost:8080 > /dev/null"

 to highlight any differences in the output of the command and the 

 flag means run the command every second. So every second we will see the changes between the current run of the command and the previous run. We remove the 

 from above because redirecting stderr into stdout (that's what 

But this still requires someone to stare at command output, which is not ideal. Instead let us use something prebuilt and automated which alerts when things change.

 is an open-source project from Google that focuses on regularly running checks against targets to determine if they are healthy, and then returning that data to your preferred monitoring service.

Cloudprober can output its results to a variety of monitoring programs, and can send requests using a variety of protocols. Each "probe" is defined as a combination of 

You can read the documentation for all of the different functionality, but the idea here is instead of doing something manually, we have a service that regularly sends requests and validates the responses.

probe {
  name: "localhost"
  type: HTTP
  targets {
    host_names: "localhost:8080"
  }
  interval_msec: 1000  # 1s
  timeout_msec: 500    # 0.5s
}

Here is a basic config for Cloudprober which sends a request to 

 every second. Right now this won't alert or upload the data it records anywhere. We'll talk in a bit about picking a monitoring service, in the section on white box monitoring, but if you were to pick Google Cloud Monitoring, you'd add something like this to your config:

This will add a lot of metrics with the prefix 

 to Cloud Monitoring in the project you have installed this service in, or that you have set as the default project with the 

Running health checks on a schedule is a common feature that many companies offer for free or cheaply. Cloudprober lets you run all sorts of arbitrary checks, but if you just need HTTP-based checks, then there are many options!

---
title: "White Box Monitoring vs Black Box Monitoring"
# description: desc
# privateVideoUrl: url
---

# Two types of monitoring

Observability is a popular term for the overarching work of making sure you can tell what is going on with your service. In the world of observability, there are two ways to observe your system: **black box monitoring** and **white box monitoring**. Black box monitoring is the idea that you have no control over the system, and so you examine it from the outside. White box monitoring is the idea that you have full control of the system, and you can instrument internal pieces with metrics, logs and traces to understand how things are working.

### Black box monitoring

As we said, black box monitoring is examining from the outside. It is often also called _synthetic monitoring_ or _behavioral testing_. We are going to generate a synthetic event, send it through our system and see what happens. This differs from the integration testing that we discussed in Chapter 3 because we are sending our synthetic event to our production environment, instead of a test environment. That being said, this is essentially testing, because we have a predefined synthetic input, and we will have a test that will validate the output.

Synthetic events for web systems tend to be HTTP requests, which means the easiest synthetic event generator in this case is the command line application `curl`. Curl has been around since 1997, and is an essential tool. The command flags I use most frequently are `-v`, `-s`, and `-L`. Together you can watch how a request works.

    $ curl -svL http://google.com

-   `-s` is for silent, and it hides any progress bars.
-   `-v` is for verbose, and it outputs the headers sent and received to STDERR.
-   `-L` is for location, and it has curl follow any HTTP redirects that are returned.

So if your server is running locally on port 8080, you could call `curl -svL localhost:8080 > /dev/null 2>&1 | grep HTTP` every second and watch the responses come in. For reference, `> /dev/null 2>&1` will ignore any errors `curl` would otherwise output, and `| grep HTTP` will only show us output lines that include the string `HTTP`.

     > GET / HTTP/1.1
     < HTTP/1.1 200 OK

This would, of course, get boring. So instead you could add the `watch` command, which runs commands every n seconds.

    $ watch -d -n 1 "curl -svL localhost:8080 > /dev/null"

The `-d` flag tells `watch` to highlight any differences in the output of the command and the `-n 1` flag means run the command every second. So every second we will see the changes between the current run of the command and the previous run. We remove the `2>&1 | grep HTTP` from above because redirecting stderr into stdout (that's what `2>&1` does) messes with watch.

But this still requires someone to stare at command output, which is not ideal. Instead let us use something prebuilt and automated which alerts when things change.

#### Cloudprober

[Cloudprober](https://github.com/google/cloudprober) is an open-source project from Google that focuses on regularly running checks against targets to determine if they are healthy, and then returning that data to your preferred monitoring service.

Cloudprober can output its results to a variety of monitoring programs, and can send requests using a variety of protocols. Each "probe" is defined as a combination of **type, interval, targets**, and **validators**. To define a new probe:

-   choose a **type**, such as HTTP
-   define a set of **targets**, like localhost:8080
-   decide the **intervals** - how frequently you want to run your test (for example every 1000 ms)
-   write a **validation** that determines if your request was good (for example, response status code equals 200)

You can read the documentation for all of the different functionality, but the idea here is instead of doing something manually, we have a service that regularly sends requests and validates the responses.

```go { actualFilename=./protected/code/cloudprober/localhost.cfg endChar=150 endLine=10 startChar=0 startLine=1 statedFilename=a basic cloudprober config url=https&#x3A;//gitlab.com/fullstackio/books/reliable-servers-with-go-book/module_05/lesson_05.01/protected/code/cloudprober/localhost.cfg }
probe {
  name: "localhost"
  type: HTTP
  targets {
    host_names: "localhost:8080"
  }
  interval_msec: 1000  # 1s
  timeout_msec: 500    # 0.5s
}
```

Here is a basic config for Cloudprober which sends a request to <http://localhost:8080> every second. Right now this won't alert or upload the data it records anywhere. We'll talk in a bit about picking a monitoring service, in the section on white box monitoring, but if you were to pick Google Cloud Monitoring, you'd add something like this to your config:

    surfacer {
      type: STACKDRIVER
    }

This will add a lot of metrics with the prefix `custom.googleapis.com/cloudprober/http/` to Cloud Monitoring in the project you have installed this service in, or that you have set as the default project with the `gcloud` command.

#### Hosted service options

Running health checks on a schedule is a common feature that many companies offer for free or cheaply. Cloudprober lets you run all sorts of arbitrary checks, but if you just need HTTP-based checks, then there are many options!

-   [Pingdom](https://www.pingdom.com/) - An old standby that has been around forever. They used to have a very generous free plan. These days they charge, but have more features.
-   [Google Cloud Monitoring Uptime Checks](https://cloud.google.com/monitoring/uptime-checks) - Free for Google Cloud Platform customers, but limits checks to once a minute.
-   [Apex Ping](https://apex.sh/ping/) - A small company, but with a comprehensive feature suite similar to Pingdom's.
-   [Datadog Uptime Checks](https://www.datadoghq.com/dg/apm/synthetics/uptime-checks/) - Datadog is a large monitoring company offering configurable synthetic uptime checks.
-   and many more!