Overcoming Challenges with Multiple Repositories - A Deep Dive into Package Management and Workspaces

Challenges in managing a large number of repositories

Project Source Code

Get the project source code below, and follow along with the lesson material.

Download Project Source Code

To set up the project on your local machine, please follow the directions provided in the README.md file. If you run into any issues with running the project source code, then feel free to reach out to the author in the course's Discord channel.

Table of Contents
  • |

Lesson Transcript

  • [00:00 - 00:15] In theory, the workflow of independently developing and publishing packages, then upgrading dependencies and integrating them seamlessly, seems straightforward. Each package operates autonomously, allowing for independent testing and deployment, as it should be.

  • [00:16 - 00:28] However, reality often proves more challenging. Mistakes happen, and sometimes the implementation details aren't sufficiently defined, leading to the need for adjustments after integration has begun.

  • [00:29 - 00:39] This back-end thought can be particularly frustrating, especially when managing a project with numerous packages. Even if things progress smoothly, the release process tends to become slow and laborious.

  • [00:40 - 00:50] One of the primary challenges when dealing with multiple repositories is the inefficiency it produces into our workflow. Let's delve into some of these challenges.

  • [00:51 - 01:04] Slow release process. Updating a dependency within a core package triggers a cascade of pull requests across various repositories to ensure all dependencies are updated accordingly.

  • [01:05 - 01:13] The sequential process can lead to prolonged release cycles, worsening the situation if a rollback is required. Redundant testing.

  • [01:14 - 01:32] Each package typically requires testing its integration with dependencies, leading to duplicated integration tests scattered across repositories. Moreover, heavier end-to-end tests essential for comprehensive testing may also be duplicated across repositories, complicating the continuous integration process further.

  • [01:33 - 01:42] Duplication of resources. We often find ourselves duplicating configurations, development dependencies, and tooling across repositories.

  • [01:43 - 01:54] This redundancy extends to continuous integration configurations. NPM commands, lead to settings, and even documentation, resulting in excessive reliance on copy-pasting, which is far from ideal.

  • [01:55 - 02:04] Prototyping and debugging challenges. Rapid prototyping for significant system changes becomes cumbersome when limited to modifying one package at a time.

  • [02:05 - 02:23] While incremental changes are desirable for controlled deployment, there are instances where shift concept validation or simultaneous debugging across multiple packages is necessary, posing a significant challenge in such segmented environment. Let's reflect on our past attempts to address certain issues.

  • [02:24 - 02:49] As revisiting them may shed light on whether our current tools adequately resolve these problems, or the years we've devised various workarounds to mitigate specific issues, but often solving one problem inadvertently exacerbates another. Consider the concept of a pseudo-work space, where dependencies between packages within a repository are managed without requiring installation from remote resources.

  • [02:50 - 03:03] This idea is akin to what is now known as workspace in tools like NPM and PNPM. In the absence of dedicated workspace tools in the past, we experimented with several approaches, each with its drawbacks.

  • [03:04 - 03:12] Pseudo-work space using local parts. With NPM version 2, local directly parts in package.json became viable.

  • [03:13 - 03:20] Although an improvement, this method still caused issues, including undesirable changes to package lock files. Pseudo-work space using NPM link.

  • [03:21 - 03:37] NPM's link command available for years allowed local package linking, but necessitated manual execution for each package and lacked configuration file support. Global same link creation often led to numerous issues, publishing to a private registry.

  • [03:38 - 03:53] Projects like Fedachio provided a local NPM registry, facilitating local package publication. However, managing multiple packages, versioning, and integration in such steps often proved cumbersome, particularly with complex CI/CD pipelines.

  • [03:54 - 04:05] All writing code in Nord modules. On occasion, resorting to directly overriding packages in the Nord modules, folder was necessary, even though far from an ideal workflow using Babel earlier.

  • [04:06 - 04:19] Leveraging Babel or TypeScript earlier allowed for local workspace imports, reducing reliance on NPM download packages. While effective, this method requires manual configuration per package and added build overhead.

  • [04:20 - 04:37] These methods primarily aim to address local dependency management challenges in a mono-repo context, but overlooked another issue. In a mono-repo setup, testing everything upon code promotion becomes a significant overhead unless we devise a method for intelligently amplifying modified packages and their dependence.

  • [04:38 - 04:54] In this initial module, we looked at what problems we deal in modern day to mitigate issues with multiple repositories in an enterprise environment. In the next module, we will dive into P&P and workspaces and how they can help us mitigate these issues.

  • [04:55 - 04:56] See you in the next module.

In theory, the workflow of independently developing and publishing packages, then upgrading dependencies and integrating them seamlessly, seems straightforward. Each package operates autonomously, allowing for independent testing and deployment, as it should be. However, reality often proves more challenging. Mistakes happen, and sometimes the implementation details aren't sufficiently defined, leading to the need for adjustments after integration has begun. This back-end thought can be particularly frustrating, especially when managing a project with numerous packages. Even if things progress smoothly, the release process tends to become slow and laborious. One of the primary challenges when dealing with multiple repositories is the inefficiency it produces into our workflow. Let's delve into some of these challenges. Slow release process. Updating a dependency within a core package triggers a cascade of pull requests across various repositories to ensure all dependencies are updated accordingly. The sequential process can lead to prolonged release cycles, worsening the situation if a rollback is required. Redundant testing. Each package typically requires testing its integration with dependencies, leading to duplicated integration tests scattered across repositories. Moreover, heavier end-to-end tests essential for comprehensive testing may also be duplicated across repositories, complicating the continuous integration process further. Duplication of resources. We often find ourselves duplicating configurations, development dependencies, and tooling across repositories. This redundancy extends to continuous integration configurations. NPM commands, lead to settings, and even documentation, resulting in excessive reliance on copy-pasting, which is far from ideal. Prototyping and debugging challenges. Rapid prototyping for significant system changes becomes cumbersome when limited to modifying one package at a time. While incremental changes are desirable for controlled deployment, there are instances where shift concept validation or simultaneous debugging across multiple packages is necessary, posing a significant challenge in such segmented environment. Let's reflect on our past attempts to address certain issues. As revisiting them may shed light on whether our current tools adequately resolve these problems, or the years we've devised various workarounds to mitigate specific issues, but often solving one problem inadvertently exacerbates another. Consider the concept of a pseudo-work space, where dependencies between packages within a repository are managed without requiring installation from remote resources. This idea is akin to what is now known as workspace in tools like NPM and PNPM. In the absence of dedicated workspace tools in the past, we experimented with several approaches, each with its drawbacks. Pseudo-work space using local parts. With NPM version 2, local directly parts in package.json became viable. Although an improvement, this method still caused issues, including undesirable changes to package lock files. Pseudo-work space using NPM link. NPM's link command available for years allowed local package linking, but necessitated manual execution for each package and lacked configuration file support. Global same link creation often led to numerous issues, publishing to a private registry. Projects like Fedachio provided a local NPM registry, facilitating local package publication. However, managing multiple packages, versioning, and integration in such steps often proved cumbersome, particularly with complex CI/CD pipelines. All writing code in Nord modules. On occasion, resorting to directly overriding packages in the Nord modules, folder was necessary, even though far from an ideal workflow using Babel earlier. Leveraging Babel or TypeScript earlier allowed for local workspace imports, reducing reliance on NPM download packages. While effective, this method requires manual configuration per package and added build overhead. These methods primarily aim to address local dependency management challenges in a mono-repo context, but overlooked another issue. In a mono-repo setup, testing everything upon code promotion becomes a significant overhead unless we devise a method for intelligently amplifying modified packages and their dependence. In this initial module, we looked at what problems we deal in modern day to mitigate issues with multiple repositories in an enterprise environment. In the next module, we will dive into P&P and workspaces and how they can help us mitigate these issues. See you in the next module.