Your First Node API

Async

A Complete Server: Getting Started

A Complete Server: Persistence

A Complete Server: Authentication

A Complete Server: Deployment

Command Line Interfaces

Testing Node.js Applications

Bonus: Persistence with PostgreSQL

Changelog

Enter your name and email address below, then click the **"Send my free chapter"** button to get started

The Complete Guide to Building Production Apps with Node.js

Fullstack Node.js

Node.js was created in reaction to slow web servers in Ruby and other dynamic languages at that time.

These servers were slow because they were only capable of handling a single request at a time. Any work that involved I/O (e.g. network or file system access) was "blocking". The program would not be able to perform any work while waiting on these blocking resources.

Node.js is able to handle many requests concurrently because it is non-blocking by default. Node.js can continue to perform work while waiting on slow resources.

The simplest, and most common form of asynchronous execution within Node.js is the callback. A callback is a way to specify that "after X happens, do Y". Typically, "X" will be some form of slow I/O (e.g. reading a file), and "Y" will be work that incorporates the result (e.g. processing data from that file).

If you're familiar with JavaScript in the browser, you'll find a similar pattern all over the place. For example:

window.addEventListener('resize', () => console.log('window has been resized!'))

​
To translate this back into words: "After the window is resized, print 'window has been resized!'"

setTimeout(() => console.log('hello from the past'), 5000)

​
Similarly, this can be translated to "After 5 seconds, print 'hello from the past'"
​
This can be confusing for most people the first time they encounter it. This is understandable because it requires thinking about multiple points in time at once.

In other languages, we expect work to be performed in the order it is written in the file. However in JavaScript, we can make the following lines print in the reverse order from how they are written:

const tenYears = 10 * 365 * 24 * 60 * 60 * 1000
setTimeout(() => console.log('hello from the past'), tenYears)
console.log('hello from the present')

​
If we were to run the above code, you would immediately see "hello from the present," and 10 years later, you would see "hello from the past."

 is non-blocking, we don't need to wait 10 years to print "hello from the present" -- it happens immediately after.

Let's take a closer look at this non-blocking behavior. We'll set up both an interval and a timeout. Our interval will print the running time of our script in seconds, and our timeout will print "hello from the past" after 5.5 seconds (and then exit so that we don't count forever):

let count = 0
setInterval(() => console.log(`${++count} mississippi`), 1000)

setTimeout(() => {
  console.log('hello from the past!')
  process.exit()
}, 5500)

 is a globally available object in Node.js. We don't need to use 

 to access it. In addition to providing us the 

 method, it's also useful for getting command-line arguments with 

. We'll cover these and more in later chapters.

 we should expect to see something like this:

❯ node 01-set-timeout.js
1 mississippi
2 mississippi
3 mississippi
4 mississippi
5 mississippi
hello from the past!

Our script dutifully counts each second, until our timeout function executes after 5.5 seconds, printing "hello from the past!" and exiting the script.

Let's compare this to what would happen if instead of using a non-blocking 

let count = 0
setInterval(() => console.log(`${++count} mississippi`), 1000)

setTimeoutSync(5500)
console.log('hello from the past!')
process.exit()

function setTimeoutSync (ms) {
  const t0 = Date.now()
  while (Date.now() - t0 < ms) {}
}

 function that will block execution for the specified number of milliseconds. This will behave more similarly to other blocking languages. However, if we run it, we'll see a problem:

❯ node 02-set-timeout-sync.js
hello from the past!

In our previous example, Node.js was able to perform two sets of instructions concurrently. While we were waiting on the "hello from the past!" message, we were seeing the seconds get counted. However, in this example, Node.js is blocked and is never able to count the seconds.

Node.js is non-blocking by default, but as we can see, it's still possible to block. Node.js is single-threaded, so long running loops like the one in 

 will prevent other work from being performed (e.g. our interval function to count the seconds). In fact, if we were to use 

 in our API server in chapter 1, our server would not be able to respond to any browser requests while that function is active!

In this example, our long-running loop is intentional, but in the future we'll be careful not to unintentionally create blocking behavior like this. Node.js is powerful because of its ability to handle many requests concurrently, but it's unable to do that when blocked.

Of course, this works the same with JavaScript in the browser. The reason why async functions like 

 exist is so that we don't block the execution loop and freeze the UI. Our 

 function would be equally problematic in a browser environment.

What we're really talking about here is having the ability to perform tasks on different timelines. We'll want to perform some tasks sequentially and others concurrently. Some tasks should be performed immediately, and others should be performed only after some criteria has been met in the future.

JavaScript and Node.js may seem strange because they try not to block by running everything sequentially in a single timeline. However, we'll see that this is gives us a lot of power to efficiently program tasks involving multiple timelines.

In the next sections we'll cover some different ways that Node.js allows us to do this using asynchronous execution. Callback functions like the one seen in 

 are the most common and straightforward, but we also have other techniques. These include promises, async/await, event emitters, and streams.

Node.js-style callbacks are very similar to how we would perform asynchronous execution in the browser and are just a slight variation on our 

Interacting with the filesystem is extremely slow relative to interacting with system memory or the CPU. This slowness makes it conceptually similar to 

While loading a small file may only take two milliseconds to complete, that's still a really long time -- enough to do over 10,000 math operations. Node.js provides us asynchronous methods to perform these tasks so that our applications can continue to perform operations while waiting on I/O and other slow tasks.

Here's what it looks like to read a file using the core 

 module has methods that allow us to interact with the filesystem. Most often we'll use this to read and write to files, get file information such as size and modified time, and see directory listings. In fact, we've already used it in the first chapter to send static files to the browser.

const fs = require('fs')

const filename = '03-read-file-callback.js'

fs.readFile(filename, (err, fileData) => {
  if (err) return console.error(err)

  console.log(`${filename}: ${fileData.length}`)
})

This difference in ordering highlights an important Node.js convention. In Node.js official APIs (and most third-party libraries) the callback is always the last argument.

The second thing to notice is the order of the arguments in the callback itself:

fs.readFile(filename, (err, fileData) => {

Here we provide an anonymous function as the callback to 

, and our anonymous function accepts two arguments: 

. This shows off another important Node.js convention: the error (or 

 if no error occurred) is the first argument when executing a provided callback.

This convention signifies the importance of error handling in Node.js. The error is the first argument because we are expected to check and handle the error first before moving on.

In this example, that means first checking to see if the error exists and if so, printing it out with 

 and skipping the rest of our function by returning early. Only if 

 is falsy, do we print the filename and file size:

fs.readFile(filename, (err, fileData) => {
  if (err) return console.error(err)

  console.log(`${filename}: ${fileData.length}`)
})

❯ node 03-read-file-callback.js
03-read-file-callback.js: 204

To trigger our error handling, change the filename to something that doesn't exist and you'll see something like this:

❯ node 03-read-file-callback-error.js
{ [Error: ENOENT: no such file or directory, open 'does-not-exist.js']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'does-not-exist.js' }

If you were to comment out our line for error handling, you would see a different error: 

TypeError: Cannot read property 'length' of undefined

. Unlike the error above, this would would crash our script.

We now know how basic async operations work in Node.js. If instead of reading a file, we wanted to get a directory list, it would work similarly. We would call 

, and because of Node.js convention, we could guess that the first argument is the directory path and the last argument is a callback. Furthermore, we know the callback that we pass should expect 

 as the first argument, and the directory listing as the second argument.

fs.readdir(directoryPath, (err, fileList) => {
  if (err) return console.error(err)

  console.log(fileList)
})

 are capitalized differently, it's because 

 and most other methods are higher-level wrappers and conform to the typical JavaScript camelCase convention.

Let's now move beyond using single async methods in isolation. In a typical real-world app, we'll need to use multiple async calls together, where we use the output from one as the input for others.

In the previous section, we learned how to perform asynchronous actions in series by using a callback to wait until an asynchronous action has completed. In this section, we'll not only perform asynchronous actions in series, but we will also perform a group of actions in parallel.

Now that we know how to read files and directories. Let's combine these to so that we can first get a directory list, and then read each file on that list. In short, our program will:

---
code: https://gitlab.com/fullstackio/books/newline-course-apps/david-guttman-fullstack-node.js-book-app/-/tree/main/code/src/02-async?ref_type=heads
---

# Async

Node.js was created in reaction to slow web servers in Ruby and other dynamic languages at that time.

These servers were slow because they were only capable of handling a single request at a time. Any work that involved I/O (e.g. network or file system access) was "blocking". The program would not be able to perform any work while waiting on these blocking resources.

Node.js is able to handle many requests concurrently because it is non-blocking by default. Node.js can continue to perform work while waiting on slow resources.

The simplest, and most common form of asynchronous execution within Node.js is the callback. A callback is a way to specify that "after X happens, do Y". Typically, "X" will be some form of slow I/O (e.g. reading a file), and "Y" will be work that incorporates the result (e.g. processing data from that file).

If you're familiar with JavaScript in the browser, you'll find a similar pattern all over the place. For example:

```js
window.addEventListener('resize', () => console.log('window has been resized!'))
```

​
To translate this back into words: "After the window is resized, print 'window has been resized!'"

Here's another example:
​

```js
setTimeout(() => console.log('hello from the past'), 5000)
```

​
Similarly, this can be translated to "After 5 seconds, print 'hello from the past'"
​
This can be confusing for most people the first time they encounter it. This is understandable because it requires thinking about multiple points in time at once.

In other languages, we expect work to be performed in the order it is written in the file. However in JavaScript, we can make the following lines print in the reverse order from how they are written:

```js
const tenYears = 10 * 365 * 24 * 60 * 60 * 1000
setTimeout(() => console.log('hello from the past'), tenYears)
console.log('hello from the present')
```

​
If we were to run the above code, you would immediately see "hello from the present," and 10 years later, you would see "hello from the past."

Importantly, because `setTimeout()` is non-blocking, we don't need to wait 10 years to print "hello from the present" -- it happens immediately after.

Let's take a closer look at this non-blocking behavior. We'll set up both an interval and a timeout. Our interval will print the running time of our script in seconds, and our timeout will print "hello from the past" after 5.5 seconds (and then exit so that we don't count forever):

```javascript { actualFilename=code/src/02-async/01-set-timeout.js endChar=161 endLine=8 startChar=0 startLine=1 statedFilename=02-async/01-set-timeout.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/02-async/01-set-timeout.js }
let count = 0
setInterval(() => console.log(`${++count} mississippi`), 1000)

setTimeout(() => {
  console.log('hello from the past!')
  process.exit()
}, 5500)
```

Q> [`process`](https://nodejs.org/api/process.html) is a globally available object in Node.js. We don't need to use `require()` to access it. In addition to providing us the [`process.exit()`](https://nodejs.org/api/process.html#process_process_exit_code) method, it's also useful for getting command-line arguments with [`process.argv`](https://nodejs.org/api/process.html#process_process_argv) and environment variables with [`process.env`](https://nodejs.org/api/process.html#process_process_env). We'll cover these and more in later chapters.

 If we run this with `node 01-set-timeout.js` we should expect to see something like this:

    ❯ node 01-set-timeout.js
    1 mississippi
    2 mississippi
    3 mississippi
    4 mississippi
    5 mississippi
    hello from the past!

Our script dutifully counts each second, until our timeout function executes after 5.5 seconds, printing "hello from the past!" and exiting the script.

Let's compare this to what would happen if instead of using a non-blocking `setTimeout()`, we use a blocking `setTimeoutSync()` function:

```javascript { actualFilename=code/src/02-async/02-set-timeout-sync.js endChar=242 endLine=12 startChar=0 startLine=1 statedFilename=02-async/02-set-timeout-sync.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/02-async/02-set-timeout-sync.js }
let count = 0
setInterval(() => console.log(`${++count} mississippi`), 1000)

setTimeoutSync(5500)
console.log('hello from the past!')
process.exit()

function setTimeoutSync (ms) {
  const t0 = Date.now()
  while (Date.now() - t0 < ms) {}
}
```

We've created our own `setTimeoutSync()` function that will block execution for the specified number of milliseconds. This will behave more similarly to other blocking languages. However, if we run it, we'll see a problem:

    ❯ node 02-set-timeout-sync.js
    hello from the past!

What happened to our counting?

In our previous example, Node.js was able to perform two sets of instructions concurrently. While we were waiting on the "hello from the past!" message, we were seeing the seconds get counted. However, in this example, Node.js is blocked and is never able to count the seconds.

Node.js is non-blocking by default, but as we can see, it's still possible to block. Node.js is single-threaded, so long running loops like the one in `setTimeoutSync()` will prevent other work from being performed (e.g. our interval function to count the seconds). In fact, if we were to use `setTimeoutSync()` in our API server in chapter 1, our server would not be able to respond to any browser requests while that function is active!

In this example, our long-running loop is intentional, but in the future we'll be careful not to unintentionally create blocking behavior like this. Node.js is powerful because of its ability to handle many requests concurrently, but it's unable to do that when blocked.

Of course, this works the same with JavaScript in the browser. The reason why async functions like `setTimeout()` exist is so that we don't block the execution loop and freeze the UI. Our `setTimeoutSync()` function would be equally problematic in a browser environment.

What we're really talking about here is having the ability to perform tasks on different timelines. We'll want to perform some tasks sequentially and others concurrently. Some tasks should be performed immediately, and others should be performed only after some criteria has been met in the future.

JavaScript and Node.js may seem strange because they try not to block by running everything sequentially in a single timeline. However, we'll see that this is gives us a lot of power to efficiently program tasks involving multiple timelines.

In the next sections we'll cover some different ways that Node.js allows us to do this using asynchronous execution. Callback functions like the one seen in `setTimeout()` are the most common and straightforward, but we also have other techniques. These include promises, async/await, event emitters, and streams.

## Callbacks

Node.js-style callbacks are very similar to how we would perform asynchronous execution in the browser and are just a slight variation on our `setTimeout()` example above.

Interacting with the filesystem is extremely slow relative to interacting with system memory or the CPU. This slowness makes it conceptually similar to `setTimeout()`.

While loading a small file may only take two milliseconds to complete, that's still a really long time -- enough to do over 10,000 math operations. Node.js provides us asynchronous methods to perform these tasks so that our applications can continue to perform operations while waiting on I/O and other slow tasks.

Here's what it looks like to read a file using the core [`fs`](https://nodejs.org/api/fs.html) module:

Q> The core `fs` module has methods that allow us to interact with the filesystem. Most often we'll use this to read and write to files, get file information such as size and modified time, and see directory listings. In fact, we've already used it in the first chapter to send static files to the browser.

```javascript { actualFilename=code/src/02-async/03-read-file-callback.js endChar=204 endLine=10 startChar=0 startLine=1 statedFilename=02-async/03-read-file-callback.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/02-async/03-read-file-callback.js }
const fs = require('fs')

const filename = '03-read-file-callback.js'

fs.readFile(filename, (err, fileData) => {
  if (err) return console.error(err)

  console.log(`${filename}: ${fileData.length}`)
})
```

​
From this example we can see that `fs.readFile()` expects two arguments, `filename` and `callback`:

```js
fs.readFile(filename, callback)
```

`setTimeout()` also expects two arguments, `callback` and `delay`:

```js
setTimeout(callback, delay)
```

This difference in ordering highlights an important Node.js convention. In Node.js official APIs (and most third-party libraries) the callback is always the last argument.

The second thing to notice is the order of the arguments in the callback itself:

```javascript { actualFilename=code/src/02-async/03-read-file-callback.js endLine=5 startLine=5 statedFilename=02-async/03-read-file-callback.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/02-async/03-read-file-callback.js#L5 }
fs.readFile(filename, (err, fileData) => {
```

Here we provide an anonymous function as the callback to `fs.readFile()`, and our anonymous function accepts two arguments: `err` and `fileData`. This shows off another important Node.js convention: the error (or `null` if no error occurred) is the first argument when executing a provided callback.

This convention signifies the importance of error handling in Node.js. The error is the first argument because we are expected to check and handle the error first before moving on.

In this example, that means first checking to see if the error exists and if so, printing it out with `console.error()` and skipping the rest of our function by returning early. Only if `err` is falsy, do we print the filename and file size:

```javascript { actualFilename=code/src/02-async/03-read-file-callback.js endLine=9 startLine=5 statedFilename=02-async/03-read-file-callback.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/02-async/03-read-file-callback.js#L5 }
fs.readFile(filename, (err, fileData) => {
  if (err) return console.error(err)

  console.log(`${filename}: ${fileData.length}`)
})
```

Run this file with `node 03-read-file-callback.js` and you should see output like:

    ❯ node 03-read-file-callback.js
    03-read-file-callback.js: 204

To trigger our error handling, change the filename to something that doesn't exist and you'll see something like this:

    ❯ node 03-read-file-callback-error.js
    { [Error: ENOENT: no such file or directory, open 'does-not-exist.js']
      errno: -2,
      code: 'ENOENT',
      syscall: 'open',
      path: 'does-not-exist.js' }

If you were to comment out our line for error handling, you would see a different error: `TypeError: Cannot read property 'length' of undefined`. Unlike the error above, this would would crash our script.

We now know how basic async operations work in Node.js. If instead of reading a file, we wanted to get a directory list, it would work similarly. We would call `fs.readdir()`, and because of Node.js convention, we could guess that the first argument is the directory path and the last argument is a callback. Furthermore, we know the callback that we pass should expect `error` as the first argument, and the directory listing as the second argument.

```javascript { actualFilename=code/src/02-async/04-read-dir-callback.js endLine=9 startLine=5 statedFilename=02-async/04-read-dir-callback.js url=https&#x3A;//gitlab.com/fullstackio/books/fullstack-nodejs-book/fullstack-nodejs/code/src/02-async/04-read-dir-callback.js#L5 }
fs.readdir(directoryPath, (err, fileList) => {
  if (err) return console.error(err)

  console.log(fileList)
})
```

Q> If you're wondering why `fs.readFile()` and `fs.readdir()` are capitalized differently, it's because `readdir` is a [system call](http://man7.org/linux/man-pages/man3/readdir.3.html), and `fs.readdir()` follows its C naming convention. `fs.readFile()` and most other methods are higher-level wrappers and conform to the typical JavaScript camelCase convention.

Let's now move beyond using single async methods in isolation. In a typical real-world app, we'll need to use multiple async calls together, where we use the output from one as the input for others.

### Async in Series and Parallel

In the previous section, we learned how to perform asynchronous actions in series by using a callback to wait until an asynchronous action has completed. In this section, we'll not only perform asynchronous actions in series, but we will also perform a group of actions in parallel.

Now that we know how to read files and directories. Let's combine these to so that we can first get a directory list, and then read each file on that list. In short, our program will: