今天在思考如何在Linux服务器上不依赖PM2部署直接部署Python Web服务时，碰巧搜到的这片文章。作为一片入门教程，作者给出了非常明确的思路和操作示例，并在字里行间和文末明列出了推荐阅读材料。为推荐给各位阅读，这里我做了全文的复制，原文链接。
感谢作者Luke Bond，有机会我会将全文翻译成中文。
The Node.js community has embraced process monitoring tools such as PM2, Nodemon, and Forever, which is understandable. For example, in addition to process monitoring, PM2 also boasts features around logging and port-sharing or clustering.
However, I’m a firm believer in using the Linux init system for process monitoring. In this blog post, I’ll show you how to recreate process management, logging and clustering functionality using the Linux init system, systemd, and I’ll make the case for this being a superior approach.
Please note that I’ve no intention of casting aspersions on any of the tools I’ve mentioned. But I think gaining familiarity with Linux is important for Node.js developers; it’s important to use standard tools that are well-proven and widely understood by sysadmins everywhere.
A Note about PM2
I will be making reference to PM2 because it has become ubiquitous in the Node.js community, and therefore it will serve as most people’s frame of reference. PM2 makes it very easy to do:
- Process management
- Log management
- Port-sharing magic for Node.js applications
PM2’s ease of use is certainly one of its strongest points; it hides some of the operational realities of running services on Linux from Node.js developers. In this blog post, I’m going to show you how to do each of these three things with systemd.
An Explanation of Linux Init Systems
Although PM2 and similar tools are ubiquitous in the Node.js world, that’s not necessarily the case in other communities. Running your application with the Linux init system will ensure that it’s familiar to any Linux sysadmin. Therefore, knowing more about Linux, the operating system on which the vast majority of Node.js applications run, is very important for Node.js developers.
First, let’s run through a brief primer on what Linux init systems are.
Each Linux distribution has a master process running as PID 1 (process ID 1) that is the ancestor of all processes that run on the system. Even if an application spawns a bunch of child processes and orphans them, the init system will still be their ancestor and will clean them up.
The init system is responsible for starting and stopping services on boot. Typically, sysadmins will write init scripts to start, stop, and restart each service (e.g., databases, web servers). Basically, the Linux init system is the ultimate process monitor.
systemd is more or less the standard Linux system in the latest release of most Linux distributions, so that’s the one I’m going to cover here. It should be relatively easy to translate these concepts into another init system, such as upstart.
Creating a Sample Node.js Application
To aid explanation, I’m going to use a simple, contrived Node.js application that talks to Redis. It has one HTTP endpoint that outputs “Hello, World!” and a counter taken from Redis. It can be found here:
https://github.com/lukebond/demo-api-redis
You will also need:
- A Linux distribution running systemd
- Node.js installed
- Redis installed (but not running)
Clone the above repository to somewhere in your Linux system and run npm install.
Creating Unit Files
Next we’ll create a unit file for our Node.js service. A unit file is what systemd uses to describe a service, its configuration, how to run it, and so on. It’s a text file similar to an INI file.
Create the following text file and copy it to /etc/systemd/system/demo-api-redis@.service:
1 | [Unit] |
Remember! Modify the path on the
WorkingDirectory=line to the location where you cloned the git repository.
Now that the unit file is created and is in the correct location on your system, we need to tell systemd to reload its config to pick up the new unit file, then enable and start the service:
1 | $ systemctl daemon-reload |
Learn more about how to use
systemctlhere.
Enabling a service means that systemd will start that service automatically on boot, but it doesn’t start it now. Starting a service is required to start the service now.
Check the status of the service to see if it worked:
1 | $ systemctl status demo-api-redis@1 |
This is failing because Redis isn’t running. Let’s explore dependencies in systemd!
Exploring systemd Dependencies
We can add the Wants= directive to the [Unit] section of a unit file to declare dependencies between services. There are other directives with different semantics (e.g., Requires=) but Wants= will cause the depended-upon service (in this case, Redis) to be started when our Node.js service is started.
Your unit file should now look like this:
1 | [Unit] |
Signal systemd to reload its config:
1 | $ systemctl daemon-reload |
Ask systemd to cat the unit file just to ensure it has picked up our changes:
1 | $ systemctl cat demo-api-redis@1 |
And now restart the service. We can see that the service now works:
1 | $ systemctl restart demo-api-redis@1 |
It works because it has triggered Redis to run:
1 | $ systemctl status redis |
Process Management
The first item of PM2 functionality we’re working toward is process management. This means restarting services when they crash and when the machine reboots. Do we have this functionality yet? Let’s find out.
1 | $ systemctl status demo-api-redis@1 | grep "PID" |
So systemd is not restarting our service when it crashes, but never fear — systemd has a range of options for configuring this behavior. Adding the following to the [Service] section of our unit file will be fine for our purposes:
1 | Restart=always |
This tells systemd to always restart the service after a 500ms delay. You can configure it to give up eventually, but this should be fine for our purposes. Now reload systemd’s config and restart the service and try killing the process:
1 | $ systemctl daemon-reload |
It works! systemd is now restarting our service when it goes down. It will also start it up automatically if the machine reboots (that’s what it means to enable a service). Go ahead and reboot to prove it.
We’ve now recreated one of our three PM2 features: process management. Let’s move on to the next one.
Logging
This is the easiest of our three target features. systemd has a very powerful logging tool called journalctl. It’s a sysadmin’s Swiss Army knife of logging, and it can do anything you’ll ever need from a logging tool. No Node.js userland tool comes close.
To scroll through logs for a unit or service:
1 | $ journalctl -u demo-api-redis@1 |
To follow the same:
1 | $ journalctl -u demo-api-redis@1 -f |
You can ask for logs since the last boot:
1 | $ journalctl -u demo-api-redis@1 --boot |
Or since a specific time, in various ways:
1 | $ journalctl -u demo-api-redis@1 --since 08:00 |
You can filter by log level (console.log, console.error, etc.):
1 | $ journalctl -u demo-api-redis@1 -p err |
There is so much more you can do; it’s super powerful. This article is a great place to start to learn all about journalctl.
Multiple Instances
We’ve covered two of our three features now. The last one is port sharing, or clustering as it is often called in the Node.js world. But before we can address that, we need to be able to run multiple instances of our service.
You may have noticed that our unit file has an @ symbol in the filename, and that we’ve been referring to our service as demo-api-redis@1. The 1 after the @ symbol is the instance name (it doesn’t have to be a number). We could run two more instances of our service using something like systemctl start demo-api-redis@{2,3}, but first we need them to bind to different ports or they’ll clash.
Our sample app takes an environment variable to set the port, so we can use the instance name to give each service a unique port. Add the following additional Environment= line to the [Service] section of the unit file:
1 | Environment=LISTEN_PORT=900%i |
This will mean that demo-api-redis@1 will get port 9001, demo-api-redis@2 will get port 9002, and demo-api-redis@3 will get port 9003, leaving 9000 for our load balancer.
Once you’ve edited the unit file, you need to reload the config, check that it’s correct, start two new instances, and restart the existing one:
1 | $ systemctl daemon-reload |
We should now be able to curl each of these:
1 | $ curl localhost:900{1,2,3} |
I’m assuming a 4-core machine, so I’m running three instances, leaving one core for Redis (which is probably not necessary). Adjust this accordingly for your environment and application.
Now, on to the final part: load balancing.
Load Balancing
One could use NGINX or HAProxy to balance the traffic across the instances of our service. However, since I’m claiming that it’s super simple to replace PM2 functionality, I wanted to go with something lighter.
Balance is a tiny (few-hundred lines of C) TCP load balancer that’s fast and simple to use. For example:
1 | $ balance -f 9000 127.0.0.1:900{1,2,3} & |
The above one-liner launches balance, listening on port 9000 and balancing across ports 9001-9003. But we don’t want to run it in the foreground like this. Let’s write a unit file:
1 | $ cat /etc/systemd/system/balance.service |
Conclusion
We’ve successfully recreated the three main features of PM2 using basic Linux tools, in fact, mostly just systemd. But this is only a very basic implementation. There are a number of details I’ve overlooked for the sake of simplicity:
- SSL termination.
- Ports
9001-9003are currently bound to the public IP, not the private (this is just laziness in my Node.js sample app). - The balance unit file has hardcoded ports 9001-9003; it should be relatively easy to dynamically configure balance and send it a signal to reload config.
- I’d normally use containers so that the dependencies (e.g., Node.js version) is bundled inside the container and doesn’t need to be installed on the host.
Linux init systems such as systemd are the ultimate process monitor, and systemd in particular is so much more than that. It can do all that PM2 and similar tools can do, and then some. The tooling is far superior, it’s more mature, and it has a much larger userbase of seasoned sysadmins.
Learning to use systemd for running your Node.js applications (or any other applications for that matter) is much easier than you might think. Once you’ve spent a little time learning these concepts, I think you’ll agree that Linux is the best tool for the job. After all, you’ll need to configure the Linux init systemd to start PM2 on boot and restart it if it crashes. If you need the Linux init system to start your process monitor, why not just use it to run all your services?