An Easy Way To Compress And Download Log Files From Remote Server Using Node.js

As I've written in the article below, there is a quite easy way to connect to and control remote Linux servers with Node.js.

How To Manage Remote Servers And Databases Through SSH Tunnel With Node.js

I've shown examples to run commands on server, and executing queries on remote database using the "node-ssh" package. These kind of remote scripts makes it easy to save time managing massive servers.

This time, I want to show you how to automate file downloading from remote servers.

Collecting and analyzing log files are common tasks for server admins. Log files are often huge, so you have to compress the file, download it to local machine, decompress the file at local, and delete the compressed file from server. It's not a difficult task, but it takes time. And this task comes up repeatedly.

There are nice packages for Node.js to automate these tasks.

"node-ssh" is a package for SSH connecting, and it also have functions to upload and download files through SSH.

https://github.com/steelbrain/node-ssh

"tar" package have functions to compress and decompress files, and it works also on Windows.

https://www.npmjs.com/package/tar

Using these packages, I've wrote a script that...

1. Connect with SSH to remote server
2. Compress log file with tar command
3. Download compressed log file to local machine
4. Delete compressed file from server
5. Decompress log file on local machine

Here is the script:

Share:

Read More

How To Manage Remote Servers And Databases Through SSH Tunnel With Node.js

I work as a web programmer, and I sometimes connect to linux servers using SSH for tasks like checking and downloading logs, checking server status, updating applications, and managing databases.

These tasks don't happen so frequently, but I noticed that I'm repeating same kind of tasks every time it happens, so for this time I wrote a Node.js script to automate these remote tasks.

There are two alternatives to choose when executing programs to control remote servers: put and run the program in the remote server, or write a program that connects to the remote server and run it locally. I chose the latter this time, because I have many different servers to manage, and I didn't want to install Node.js to each servers.

Connecting To Remote Server Through SSH

"ssh2" is a popular package for using ssh from Node.js. There seems to be everything you want to do with SSH in it.

https://github.com/mscdex/ssh2

"node-ssh" is a Promise wrapper package of ssh2.

https://github.com/steelbrain/node-ssh

I'm using node-ssh this time.

npm install --save node-ssh

This is a sample script that connects to remote server and executes "ls -al" command.


If you want to use "sudo", write like this:

const res2 = await ssh.execCommand('sudo ls /var/log/httpd', {stdin: sshPassword + '\n', options: {pty: true}});

Now we can execute any command and receive results from remote server.

Connect To Remote Database Through SSH Tunnel

Next, let's try connecting and executing queries on remote databases.

We'll use "tunnel-ssh" for SSH tunneling on Node.js.
This is another wrapper package of ssh2.

https://www.npmjs.com/package/tunnel-ssh

npm install --save tunnel-ssh

This time we'll use "knex" and "pg" to control remote PostgreSQL database.

https://knexjs.org/

https://node-postgres.com/

npm install --save knex pg

Here is a template to create and connect a SSH Tunnel:

The opened tunnel closes when "tnl.close()" is called.

If you don't close it the node process keeps running, while the process is running,
you can use the SSH tunnel from other programs. This may be good when you want to create a SSH tunneling tool for other programs.

Here is a sample to select data from remote database through SSH tunnel:


You can fetch and update data from remote database just like using local database, by editing the part using knex.

This template would make your work like saving remote data to local Excel files,
bulk inserting data to remote databases, more easy.

Conclusion

I wrote about two ways to control remote servers through SSH using Node.js.

This should be helpful for automating tasks like managing many servers and remote databases at once.

Share:

Read More