Your First Node API

Async

A Complete Server: Getting Started

A Complete Server: Persistence

A Complete Server: Authentication

A Complete Server: Deployment

Command Line Interfaces

Testing Node.js Applications

Bonus: Persistence with PostgreSQL

Changelog

Enter your name and email address below, then click the **"Send my free chapter"** button to get started

The Complete Guide to Building Production Apps with Node.js

Fullstack Node.js

Once an app is running in production, it can be scary to make changes. How do we know that a new feature, fix, or refactor won't break existing functionality?

We can manually use our app the way we expect a user would to try to find bugs, but without maintaining a comprehensive checklist, it's unlikely we'd cover all possible failure points. Unfortunately, even if we did, this procedure is likely to be too time-consuming to be practical after every tweak.

If we were the only developer on a project, we might have a good sense of what's most likely to break when we make a change. This might narrow down the parts that we'd need to manually check, but this benefit is fleeting. Successful services can run in production for years, and as time passes, it's difficult to keep all those connections in our head.

Manual checks go completely off the rails once we're part of a team. Any developer who is unfamiliar with parts of the codebase or is new to the project will not know all of the ways to interact with the application or which behaviors are expected. Without some automated way to make sure that new code doesn't break things, all changes become risky and slow to implement.

If we're working on a team, even if we didn't worry about breaking 

 functionality, we'd want to verify that new code works on its own. If we're responsible for reviewing a teammate's code before merging into the production branch, it can be time-consuming to figure out exactly the new feature does and how to check that works properly in a variety of situations. Good communication can help, but it still takes time.

As you might have guessed, the key to making all of this better is to incorporate tests into our workflow. By running parts of our app in a 

 way we're able to quickly see if new features work as expected and old functionality doesn't break.

Let's start with an example: we want to create a function that tells us whether or not a number is prime.

module.exports = function isPrime (num) {}

And we can start using it to see if it works:

const isPrime = require('./01-is-prime')

console.log(2, isPrime(2))
console.log(3, isPrime(3))
console.log(4, isPrime(4))
console.log(5, isPrime(5))
console.log(9, isPrime(9))
console.log(200, isPrime(200))
console.log(1489, isPrime(1489))
console.log(2999, isPrime(2999))

Of course, when we run our example, our new module won't be very useful:

$ node 01-is-prime-example.js
2 undefined
3 undefined
4 undefined
5 undefined
9 undefined
200 undefined
1489 undefined
2999 undefined

However, that's easy enough to fix. We'll first add some functionality to our new module:

module.exports = function isPrime (num) {
  for (let den = 2; den < num; den++) {
    if (num % den === 0) return false
  }

  return true
}

$ node 02-is-prime-example.js
2 true
3 true
4 false
5 true
9 false
200 false
1489 true
2999 true

This is much better. What's great about this is that we can allow another developer to work on this module, and we can easily check to make sure that they didn't break anything. This will be very useful for refactors. As long as the example output correctly identifies each prime, we don't have to worry about a regression being introduced.

For example, let's add we have teammate who wants to add an optimization. They inform us that our module will look for divisors all the way from 2 up to the number being tested. They tell us that this is inefficient and the complexity of our algorithm is 

. They seem to know what they're talking about so we let them make a change so that we don't need to check as many numbers as before:

module.exports = function isPrime (num) {
  for (let den = 2; den <= Math.log(num); den++) {
    if (num % den === 0) return false
  }

  return true
}

Now we can just run our example to make sure that no errors were introduced:

$ node 03-is-prime-example.js
2 true
3 true
4 true
5 true
9 true
200 false
1489 true
2999 true

 are not prime numbers. Good thing we caught this issue before we deployed it to production!

This also highlights an improvement we can make to our process. We caught this because we know from experience that 

 are not prime numbers, and it was easy to see that this output had 

 though? Without checking our previous run, would we immediately know what the expected output should be?

Before we fix our module, let's improve things to make our lives easier. We're going to create a new file that will do the same thing as our example, but we're going to add some additional behavior to make it obvious when there are issues with our code. Instead of just printing out the return values of our method, we'll highlight any of them that are incorrect, and we'll print out what the result 

$ node 04-is-prime-test.js
*  2 should be prime
*  3 should be prime
X  4 should not be prime
*  5 should be prime
X  9 should not be prime
*  200 should not be prime
*  1489 should be prime
*  2999 should be prime


8 Tests
6 Passed
2 Failed


X  4 should not be prime
      expected: false
      actual: true
X  9 should not be prime
      expected: false
      actual: true

What's nice about this output is that we can see how many cases are incorrect and what we need to do to fix them. Luckily, converting our example to a test is simple:

The main difference between our example and our test is that we're storing the result of each 

 call along with the expected behavior, and after all the calls we print out the results. Additionally, if any of our results don't match what's expected, we change the exit code to 

 to indicate a failure. This is useful for when we want to this test to work in tandem with other processes. For example, we can make running 

. A failing test will exit with a code of 

, preventing further execution. To easily see this in action, we can display a message only if tests pass:

$ node 04-is-prime-test.js && echo "  This can only be seen if the tests pass "

For more information about exit codes (also called "exit status") check out the 

This is incredibly useful for deployments. We can make sure that deployment commands are executed 

Now that we have tests in place, let's fix our module. Clearly, our teammate was mistaken about using 

 to optimize our code. However, they were correct that we can reduce the search space for factors. The real answer is that we only need to search from 

Once we modify our module, we can run our tests again, and we can even check to see if we see our secret message:

$ node 04-is-prime-test-pass.js && echo "  This can only be seen if the tests pass "
*  2 should be prime
*  3 should be prime
*  4 should not be prime
*  5 should be prime
*  9 should not be prime
*  200 should not be prime
*  1489 should be prime
*  2999 should be prime


8 Tests
8 Passed
0 Failed


  This can only be seen if the tests pass 

Nice work, we've created the basics of a test framework! Now when we refactor or optimize, we can have the confidence that we aren't going to break anything.

As we know, while Node.js is single-threaded, much of its impressive performance comes from its ability to handle work concurrently. It can do this by performing synchronous work while waiting on slow IO tasks. Unfortunately, Node.js can't do the reverse: if the process is completely occupied by a synchronous task, it can't start any asynchronous work.

Let's say that we wanted to put our machine to work by finding large prime numbers, we might do something like this:

const isPrime = require('./04-is-prime-sqrt')

module.exports = function (nStart, count) {
  let n = nStart

  const primes = []

  while (primes.length < count) {
    n++
    if (isPrime(n)) primes.push(n)
  }

  return primes
}

Unfortunately, this would be problematic to use when writing a production server. For example, let's turn our 

const URL = require('url')
const http = require('http')

const isPrime = require('./04-is-prime-sqrt')
const findPrimes = require('./05-find-primes-sync')

const port = process.env.PORT || 1337
http.createServer(onRequest).listen(port)
console.log(`Prime Server listening on port ${port}`)

function onRequest (req, res) {
  const opts = URL.parse(req.url, true)

  const handlers = {
    '/is-prime': handleIsPrime,
    '/find-primes': handleFindPrimes
  }

  const handler = handlers[opts.pathname] || handleNotFound
  handler(req, res, opts)
}

function handleIsPrime (req, res, opts) {
  const n = parseFloat(opts.query.n)

  if (!isFinite(n)) return sendJSON(res, 400, { error: 'Bad Request' })

  sendJSON(res, 200, isPrime(n))
}

function handleFindPrimes (req, res, opts) {
  const nStart = parseFloat(opts.query.nStart)
  const count = parseFloat(opts.query.count)

  if (!isFinite(nStart) || !isFinite(count)) {
    return sendJSON(res, 400, { error: 'Bad Request' })
  }

  sendJSON(res, 200, findPrimes(nStart, count))
}

function handleNotFound (req, res, opts) {
  sendJSON(res, 404, { error: 'Not Found' })
}

function sendJSON (res, status, message) {
  res.writeHead(status, { 'Content-Type': 'application/json' })
  res.end(JSON.stringify(message))
}

If a client made a request to find many large primes, our server would be unable to respond to any other requests until they were found. We can test this ourselves.

 to time our requests using the following command:

curl -s -o /dev/null -w  "%{time_starttransfer}\n" $URL_TO_TEST

If we put all of this together while running our server, we can see that the 

 endpoint is pretty fast and on my machine 

$ curl -s -o /dev/null -w  "%{time_starttransfer}\n" "http://localhost:1337/is-prime?n=130"
0.005

However, if we try to find 1,000 prime numbers greater than 10,000,000,000 we have to wait a bit longer:

$ curl -s -o /dev/null -w  "%{time_starttransfer}\n" "http://localhost:1337/find-primes?nStart=1e10&count=1e4"
10.340

Now, we'll do both requests. First, we'll start the slower 

 request, and then a few seconds later we'll start the fast 

Here we can clearly see the issue. The request that should have only taken 5ms, took over 6 seconds to complete -- over 1,300x slower than it needed to be!

Let's see how we can use testing to fix this problem.

The first step is creating some tests for our 

 module. Since we want to use the testing framework we created earlier in the chapter, it's time to extract that into its own module:

---
code: https://gitlab.com/fullstackio/books/newline-course-apps/david-guttman-fullstack-node.js-book-app/-/tree/main/code/src/08-testing?ref_type=heads
---

# Testing Node.js Applications

Once an app is running in production, it can be scary to make changes. How do we know that a new feature, fix, or refactor won't break existing functionality?

We can manually use our app the way we expect a user would to try to find bugs, but without maintaining a comprehensive checklist, it's unlikely we'd cover all possible failure points. Unfortunately, even if we did, this procedure is likely to be too time-consuming to be practical after every tweak.

If we were the only developer on a project, we might have a good sense of what's most likely to break when we make a change. This might narrow down the parts that we'd need to manually check, but this benefit is fleeting. Successful services can run in production for years, and as time passes, it's difficult to keep all those connections in our head.

Manual checks go completely off the rails once we're part of a team. Any developer who is unfamiliar with parts of the codebase or is new to the project will not know all of the ways to interact with the application or which behaviors are expected. Without some automated way to make sure that new code doesn't break things, all changes become risky and slow to implement.

If we're working on a team, even if we didn't worry about breaking _existing_ functionality, we'd want to verify that new code works on its own. If we're responsible for reviewing a teammate's code before merging into the production branch, it can be time-consuming to figure out exactly the new feature does and how to check that works properly in a variety of situations. Good communication can help, but it still takes time.

As you might have guessed, the key to making all of this better is to incorporate tests into our workflow. By running parts of our app in a _controlled_ way we're able to quickly see if new features work as expected and old functionality doesn't break.

## Testing 101

Let's start with an example: we want to create a function that tells us whether or not a number is prime.

We start with a placeholder module:

```javascript { actualFilename=code/src/08-testing/01-is-prime.js endLine=2 startLine=1 statedFilename=08-testing/01-is-prime.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/01-is-prime.js }
module.exports = function isPrime (num) {}
```

And we can start using it to see if it works:

```javascript { actualFilename=code/src/08-testing/01-is-prime-example.js endLine=11 startLine=1 statedFilename=08-testing/01-is-prime-example.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/01-is-prime-example.js }
const isPrime = require('./01-is-prime')

console.log(2, isPrime(2))
console.log(3, isPrime(3))
console.log(4, isPrime(4))
console.log(5, isPrime(5))
console.log(9, isPrime(9))
console.log(200, isPrime(200))
console.log(1489, isPrime(1489))
console.log(2999, isPrime(2999))
```

Of course, when we run our example, our new module won't be very useful:

    $ node 01-is-prime-example.js
    2 undefined
    3 undefined
    4 undefined
    5 undefined
    9 undefined
    200 undefined
    1489 undefined
    2999 undefined

However, that's easy enough to fix. We'll first add some functionality to our new module:

```javascript { actualFilename=code/src/08-testing/02-is-prime.js endLine=8 startLine=1 statedFilename=08-testing/02-is-prime.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/02-is-prime.js }
module.exports = function isPrime (num) {
  for (let den = 2; den < num; den++) {
    if (num % den === 0) return false
  }

  return true
}
```

Then, we'll run our example again:

    $ node 02-is-prime-example.js
    2 true
    3 true
    4 false
    5 true
    9 false
    200 false
    1489 true
    2999 true

This is much better. What's great about this is that we can allow another developer to work on this module, and we can easily check to make sure that they didn't break anything. This will be very useful for refactors. As long as the example output correctly identifies each prime, we don't have to worry about a regression being introduced.

For example, let's add we have teammate who wants to add an optimization. They inform us that our module will look for divisors all the way from 2 up to the number being tested. They tell us that this is inefficient and the complexity of our algorithm is `O(n)` when it should be `O(log n)`. They seem to know what they're talking about so we let them make a change so that we don't need to check as many numbers as before:

```javascript { actualFilename=code/src/08-testing/03-is-prime-log.js endLine=8 startLine=1 statedFilename=08-testing/03-is-prime-log.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/03-is-prime-log.js }
module.exports = function isPrime (num) {
  for (let den = 2; den <= Math.log(num); den++) {
    if (num % den === 0) return false
  }

  return true
}
```

Now we can just run our example to make sure that no errors were introduced:

    $ node 03-is-prime-example.js
    2 true
    3 true
    4 true
    5 true
    9 true
    200 false
    1489 true
    2999 true

That's not right... `4` and `9` are not prime numbers. Good thing we caught this issue before we deployed it to production!

This also highlights an improvement we can make to our process. We caught this because we know from experience that `4` and `9` are not prime numbers, and it was easy to see that this output had `true` when it should have been `false`. What about `1489` and `2999` though? Without checking our previous run, would we immediately know what the expected output should be?

Before we fix our module, let's improve things to make our lives easier. We're going to create a new file that will do the same thing as our example, but we're going to add some additional behavior to make it obvious when there are issues with our code. Instead of just printing out the return values of our method, we'll highlight any of them that are incorrect, and we'll print out what the result _should_ be. Here's what that will look like:

    $ node 04-is-prime-test.js
    *  2 should be prime
    *  3 should be prime
    X  4 should not be prime
    *  5 should be prime
    X  9 should not be prime
    *  200 should not be prime
    *  1489 should be prime
    *  2999 should be prime


    8 Tests
    6 Passed
    2 Failed


    X  4 should not be prime
          expected: false
          actual: true
    X  9 should not be prime
          expected: false
          actual: true

What's nice about this output is that we can see how many cases are incorrect and what we need to do to fix them. Luckily, converting our example to a test is simple:

```javascript { actualFilename=code/src/08-testing/04-is-prime-test.js endLine=48 startLine=1 statedFilename=08-testing/04-is-prime-test.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/04-is-prime-test.js }
const assert = require("assert").strict;

const isPrime = require("./03-is-prime-log");

const results = [];

isEqual(isPrime(2), true, "2 should be prime");
isEqual(isPrime(3), true, "3 should be prime");
isEqual(isPrime(4), false, "4 should not be prime");
isEqual(isPrime(5), true, "5 should be prime");
isEqual(isPrime(9), false, "9 should not be prime");
isEqual(isPrime(200), false, "200 should not be prime");
isEqual(isPrime(1489), true, "1489 should be prime");
isEqual(isPrime(2999), true, "2999 should be prime");

finish();

function isEqual(actual, expected, msg) {
  try {
    assert.equal(actual, expected);
    results.push({ msg, expected, actual, error: false });
  } catch (error) {
    results.push({ msg, expected, actual, error });
  }
}

function finish() {
  const fails = results.filter(r => r.error);

  results.forEach(r => {
    const icon = r.error ? "\u{274C}" : "\u{2705}";
    console.log(`${icon}  ${r.msg}`);
  });

  console.log("\n");
  console.log(`${results.length} Tests`);
  console.log(`${results.length - fails.length} Passed`);
  console.log(`${fails.length} Failed`);
  console.log("\n");

  fails.forEach(f => {
    console.log(`\u{274C}  ${f.msg}`);
    console.log(f.error.message);
  });

  process.exitCode = fails.length ? 1 : 0;
}
```

The main difference between our example and our test is that we're storing the result of each `isPrime()` call along with the expected behavior, and after all the calls we print out the results. Additionally, if any of our results don't match what's expected, we change the exit code to `1` to indicate a failure. This is useful for when we want to this test to work in tandem with other processes. For example, we can make running `git push` conditional on our tests passing:

    $ node 04-is-prime-test.js && git push

Because `git push` follows `&&`, it will only run if `node 04-is-prime-test.js` exits with a code of `0`. A failing test will exit with a code of `1`, preventing further execution. To easily see this in action, we can display a message only if tests pass:

    $ node 04-is-prime-test.js && echo "  This can only be seen if the tests pass "

I> For more information about exit codes (also called "exit status") check out the [Exit Status](https://en.wikipedia.org/wiki/Exit_status) page on Wikipedia.

This is incredibly useful for deployments. We can make sure that deployment commands are executed _only_ if tests pass.

Now that we have tests in place, let's fix our module. Clearly, our teammate was mistaken about using `Math.log()` to optimize our code. However, they were correct that we can reduce the search space for factors. The real answer is that we only need to search from `2` up to the _square root_ of the number being tested[^1].

Once we modify our module, we can run our tests again, and we can even check to see if we see our secret message:

    $ node 04-is-prime-test-pass.js && echo "  This can only be seen if the tests pass "
    *  2 should be prime
    *  3 should be prime
    *  4 should not be prime
    *  5 should be prime
    *  9 should not be prime
    *  200 should not be prime
    *  1489 should be prime
    *  2999 should be prime


    8 Tests
    8 Passed
    0 Failed


      This can only be seen if the tests pass 

Nice work, we've created the basics of a test framework! Now when we refactor or optimize, we can have the confidence that we aren't going to break anything.

## Async Tests

As we know, while Node.js is single-threaded, much of its impressive performance comes from its ability to handle work concurrently. It can do this by performing synchronous work while waiting on slow IO tasks. Unfortunately, Node.js can't do the reverse: if the process is completely occupied by a synchronous task, it can't start any asynchronous work.

Let's say that we wanted to put our machine to work by finding large prime numbers, we might do something like this:

```javascript { actualFilename=code/src/08-testing/05-find-primes-sync.js endLine=15 startLine=1 statedFilename=08-testing/05-find-primes-sync.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/05-find-primes-sync.js }
const isPrime = require('./04-is-prime-sqrt')

module.exports = function (nStart, count) {
  let n = nStart

  const primes = []

  while (primes.length < count) {
    n++
    if (isPrime(n)) primes.push(n)
  }

  return primes
}
```

Unfortunately, this would be problematic to use when writing a production server. For example, let's turn our `isPrime()` and `findPrimes()` methods into an API:

```javascript { actualFilename=code/src/08-testing/05-prime-server.js endLine=50 startLine=1 statedFilename=08-testing/05-prime-server.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/05-prime-server.js }
const URL = require('url')
const http = require('http')

const isPrime = require('./04-is-prime-sqrt')
const findPrimes = require('./05-find-primes-sync')

const port = process.env.PORT || 1337
http.createServer(onRequest).listen(port)
console.log(`Prime Server listening on port ${port}`)

function onRequest (req, res) {
  const opts = URL.parse(req.url, true)

  const handlers = {
    '/is-prime': handleIsPrime,
    '/find-primes': handleFindPrimes
  }

  const handler = handlers[opts.pathname] || handleNotFound
  handler(req, res, opts)
}

function handleIsPrime (req, res, opts) {
  const n = parseFloat(opts.query.n)

  if (!isFinite(n)) return sendJSON(res, 400, { error: 'Bad Request' })

  sendJSON(res, 200, isPrime(n))
}

function handleFindPrimes (req, res, opts) {
  const nStart = parseFloat(opts.query.nStart)
  const count = parseFloat(opts.query.count)

  if (!isFinite(nStart) || !isFinite(count)) {
    return sendJSON(res, 400, { error: 'Bad Request' })
  }

  sendJSON(res, 200, findPrimes(nStart, count))
}

function handleNotFound (req, res, opts) {
  sendJSON(res, 404, { error: 'Not Found' })
}

function sendJSON (res, status, message) {
  res.writeHead(status, { 'Content-Type': 'application/json' })
  res.end(JSON.stringify(message))
}
```

If a client made a request to find many large primes, our server would be unable to respond to any other requests until they were found. We can test this ourselves.

We'll use `curl` to time our requests using the following command:

    curl -s -o /dev/null -w  "%{time_starttransfer}\n" $URL_TO_TEST

We use the following flags:

-   `-s` silences the progress output
-   `-o /dev/null` will take the response data and write it to `/dev/null` which will delete/hide it
-   `-w  "%{time_starttransfer}\n"` controls the format of the output, and we specify that we're interested in how long it takes before we recieve the first byte

If we put all of this together while running our server, we can see that the `/is-prime` endpoint is pretty fast and on my machine `curl` clocks the response time at 5ms:

    $ curl -s -o /dev/null -w  "%{time_starttransfer}\n" "http://localhost:1337/is-prime?n=130"
    0.005

However, if we try to find 1,000 prime numbers greater than 10,000,000,000 we have to wait a bit longer:

    $ curl -s -o /dev/null -w  "%{time_starttransfer}\n" "http://localhost:1337/find-primes?nStart=1e10&count=1e4"
    10.340

Now, we'll do both requests. First, we'll start the slower `/find-primes` request, and then a few seconds later we'll start the fast `/is-prime` request:

![Slow and fast endpoints](https://d2uusema5elisf.cloudfront.net/books/fullstack-nodejs/images/08-testing/01-server-blocking.png)

Here we can clearly see the issue. The request that should have only taken 5ms, took over 6 seconds to complete -- over 1,300x slower than it needed to be!

Let's see how we can use testing to fix this problem.

The first step is creating some tests for our `findPrimes()` module. Since we want to use the testing framework we created earlier in the chapter, it's time to extract that into its own module:

```javascript { actualFilename=code/src/08-testing/06-simple-test.js endLine=35 startLine=1 statedFilename=08-testing/06-simple-test.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/08-testing/06-simple-test.js }
const results = [];