id26.comPersonal experience with technology

To traverse firewalls that block incoming SSH connections or to access a computer with a non routable IP address, you can create an SSH tunnel. When creating a SSH tunnel, you’ll generate the tunnel on the host system and connect it to another system.

With the SSH command, you’ll use the “-R” flag, that allows reversible communications or a reverse tunnel.

This command that I normally use:

It states, create a reversible SSH tunnel from this localhost port 22 to USER@SERVER on port 10000.

 
/usr/bin/ssh -R 10000:localhost:22 USER@SERVER

You’ll need to leave this connection open otherwise the tunnel will collapse. Normally I execute a program at the other end to keep the connection alive. Some SSHd configurations will drop connections due to inactivity.

To connect to the tunnel from USER@SERVER:

use ssh to connect to your USER account on the host server via port 10000.

 
/usr/bin/ssh USER@localhost -p 10000

this will give you the login prompt at the host server. User your login information or you can also setup SSH Keys.

If a cron job executes with unmanaged warnings for data, you’ll generate LOTS of emails to your account. Here is an easy way to redirect all that extraneous data.

Just add “>> /dev/null 2>$1″ to the end of each of your cron jobs.

 
0,10,20,30,40,50 * * * * /path/to/script >> /dev/null 2>&1

Many times, I have to execute sequential commands inorder to complete a task. Below is an example of how to do this.

/path/to/command argv1 && /path/to/command argv2

I normally use this sequential order when pulling raw data from a source and then reformatting it for my needs.

15 * * * * /usr/local/bin/pullsource grab && /usr/local/bin/pullsource reformat

If you have several servers to manage, having them set with the same date and time is very important.

NTP stands for Network Time Protocol.

ntpdate – set the date and time via NTP

sudo ntpdate NTPServerAddress

or if you want to use ubuntu’s time server.

sudo ntpdate ntp.ubuntu.com

If you want to use multiple time servers, you can use a command like this. Where the new time server follows the address of your current server.

sudo ntpdate ntp.ubuntu.com time.apple.com

Links:
NTP Pool Project
Ubuntu Time Synchronization with NTP

This morning, I was working on allowing users of my web server to create web pages with in their home directory. Normally this is indicated by a URL of http://server.com/~useraccount and is activated by default. But during the Apache install process the module for modifying the user directory wasn’t activated.

List of things you’ll need to do.

1) Add the module to your virtual host.
2) Enable the module in apache2
3) Restart apache2
4) Create a web directory folder in your home folder
5) make sure your permissions are correct

Step 1, adding the module to your virtual host.

Normally in Ubuntu, your enabled virtual hosts (or virtual hosts that are current running) are located in /etc/apache2/sites-enabled/your_domain_naming_scheme. For this example, I’ll always name my virtual hosts after their qualified domain name (qdn).

sudo nano /etc/apache2/sites-enabled/server.com

With in your < virtualhost > < / virtualhost > tags, you need to add and If Module statement. An If Module states, that if the module exists in the apache lib directory, to go ahead and load it into ram. The common use of the UserDir statement is command and then directory. So in this case UserDir public_html. public_html is the directory that will need to be created in each user’s home directory in order for apache to map the URL to file systems correctly. In this case the URL of http://server.com/~dmccoy will be mapped to the file directory path of /home/dmccoy/public_html/.

        < IfModule mod_userdir.c >
                UserDir html
        < /IfModule >

a complete virtual host might look something like this.

NameVirtualHost *:80
< VirtualHost server.com:80 >
        ServerAdmin dmccoy@server.com
 
        DocumentRoot /var/www/server.com
        ServerName server.com
        ServerAlias server
 
        < Directory />
                Options FollowSymLinks
                AllowOverride None
        < /Directory>
        < Directory /var/www/server.com/ >
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
                # This directive allows us to have apache2's default start page
               # in /apache2-default/, but still have / go to the right place
              #RedirectMatch ^/$ /apache2-default/
        < /Directory >
 
        < IfModule mod_userdir.c >
                UserDir html
        < /IfModule >
 
        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        < Directory "/usr/lib/cgi-bin" >
                AllowOverride None
                Options ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        < /Directory >
 
        ErrorLog /var/log/apache2/server.com-error.log
# Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
 
        CustomLog /var/log/apache2/server.com-access.log combined
        ServerSignature On
 
   Alias /doc/ "/usr/share/doc/"
    < Directory "/usr/share/doc/" >
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    < /Directory >
< /VirtualHost >

Step 2, Once you’ve told the virtual host to load the UserDir module, you’ll need to make sure it is enabled in apache2. You can find this out by looking in the /etc/apache2/mods-enabled/ directory.

Both of these files will need to be in the mods-enabled directory.

userdir.conf
userdir.load

If they are not, make sure they are in the mods-available directory located at /etc/apache2/mods-available/. If they are in the mods-available directory, all you need to do is symbolically link them into the mods-enabled directory.

sudo ln -s /etc/apache2/mods-available/userdir.conf /etc/apache2/mods-enabled/userdir.conf
sudo ln -s /etc/apache2/mods-available/userdir.load /etc/apache2/mods-enabled/userdir.load

Step 3, You’ll need to restart apache2 in order to activate the userdir module.

sudo /etc/init.d/apache2 restart

Step 4, Now that the userdir module is active you can now add the “public_html” directory to any user that wants a website.

Normal command line steps might look something like this.

For yourself

 
cd ~
mkdir public_html
echo "test" > public_html/index.html

Or for another account

 
sudo mkdir /home/USER/public_html
sudo echo "test" > home/USER/public_html/index.html

Step 5, You should now be able to type in your browser http://server.com/~dmccoy and see “test” if you get a Forbidden error, you’ll need to change the permissions on your home directory folder and your public_html folder.

 
chmod 701 /home/dmccoy
chmod 705 /home/dmccoy/public_html

Repeat steps 4 and 5 for every account that would like web access.

Apache 2 Module mod_userdir
Apache 1.3 Module mod_userdir
Per-user web directories

When you lose/misplace/forget your root password on a RedHat-based linux distro, resetting the root password isn’t too difficult. I’ve added those instructions at the bottom in case you are interested.

In Debian distros (which includes Ubuntu) you are required to have the root password in order to change the root password (even when you boot into single user mode, as you would with Redhat versions). While this adds a level of security, it also adds a level of annoyance when you have lost your password and are in a pinch.

Not to worry though, there is a way around (albeit, not as easy).

The first steps are the same for either Redhat distros or Ubuntu. Reboot the server, and get into the boot menu by pressing ESC to enter the menu:

Next you’ll need to choose which kernel you’re going to edit (it doesn’t really matter, but I usually choose the first in the list that is *not* recovery mode):

Type ‘e’ to edit that line, and it will take you to another menu, which brings you to the line you’ll edit:

Scroll down to the ‘kernel’ like, and hit ‘e’ again. This will insert the kernel line into the grub> prompt, where you can add/edit the boot instructions. Yours will not be identical to the example below, but should have the same components:

 kernel /boot/vmlinuz-2.6.17-11-386 root=UUID=7d2d4dd5-f7fd-45ad-8bc9-51b1b284fcaa ro quiet splash

You will want to edit that to :

 grub> kernel /boot/vmlinuz-2.6.17-11-386 root=UUID=7d2d4dd5-f7fd-45ad-8bc9-51b1b284fcaa init=/bin/bash

When finished, hit enter to go back to the previous screen, and hit ‘b’ to boot that newly edited line.

After it boots, you’ll be at a shell prompt, but it isn’t writable! So any changes you make (like changing the root password) won’t hold through the reboot. So you’ll need to mount your root directory (which is simply labeled: / ) by typing the following command:

mount -o remount,rw /

Be sure to note the spacing in that command

Next you’ll need to change the root password. Since you are in the root prompt, it will change the password for user ‘root’ by default. I’ve added the root user below so you can see how to change other user passwords as well (simply replace ‘root’ with another user):

 passwd root

Enter the new password (one that you’ll remember, but is sufficiently secure).

It should reboot as it is now, but if you want to be safe you can reset the drive to read only:

mount -o remount,ro /

Then type:

 shutdown -r now

When your server reboots, you should now be able to log in as root with your newly set password. Granted, this will only work if you are using the cli (or have already set to allow root login through the settings in gnome/kde).

To perform this same task on a RedHat linux distro (RedHat, Fedora, CentOS, etc.) you follow the same instructions as above, except you add the word ‘single’ instead of ‘init=/bin/bash’ from above. Then continue to boot.

When you boot that kernel, it will drop you into a root prompt, at which point you can type passwd to change your password (with passwd, just like above). Next time you reboot your server, you will have root access with your newly formed password.

If you manage any type of databases, you know the importance of backing up your data. Whether you have file corruption from a failing hard drive, a run away script or from a hacking attempt; loosing your databases can and will break your business. If you’ve never backed up your database, now is a good time to start.

The easiest way to backup your MySQL database is with “/usr/bin/mysqldump”. You’ll need to access this application via command. Whether it is from the a console or an SSH connection.

mysqldump – a database backup program
mysqldump [options] [db_name [tbl_name ...]]

There are three general ways to invoke mysqldump:

mysqldump [options] db_name [tables]
mysqldump [options] --databases db_name1 [db_name2 db_name3...]
mysqldump [options] --all-databases

By default, the mysqldump command will dump the contents of your database directly into your terminal and not into a file. This output can be redirected anyway and to anywhere you like. But it is commonly used in this manner:

mysqldump -u [username] -p [password] [databasename] > [backup.sql]
mysqldump -u root -p PASSWORD DATABASE > DATABASE-BACKUP-DATE.sql

[username] – your database username
[password] – the password for your database
[databasename] – the name of your database
[backup.sql] – the file to which the database contents needs to be directed to

If you don’t want to use your password in the command line, you can always prompt for your password by excluding it from your command.

mysqldump -u root -p DATABASE > DATABASE-BACKUP-DATE.sql
Enter password:

If you forgot your password, you’ll get an error like this.

mysqldump: Got error: 1045: Access denied for user 'root'@'localhost' (using password: YES) when trying to connect

Or if you didn’t enter a password when one is needed, you’ll get this error.

mysqldump: Got error: 1045: Access denied for user 'root'@'localhost' (using password: NO) when trying to connect

If you’d like the convenience of having the database automatically delete the old database before re-writing the content you can use the “–add-drop-table” feature.

 mysqldump --add-drop-table -u root -p PASSWORD DATABASE > DATABASE-BACKUP-DATE.sql

If you need to only backup specific tables of the database you can use this command.

mysqldump --add-drop-table -u root -p PASSWORD DATABASE TABLE> DATABASE-TABLE-BACKUP-DATE.sql

The syntax for the command is:

mysqldump -u [username] -p [password] [databasename] [table1 table2 ....]

[tables] – This is a list of tables to backup. Each table is separated by a space.

If you are a server administrator and need to backup multiple databases at the same time, you can use the next command. You just have to be sure that the “-u” user account has permission to read those databases.

mysqldump -u root -p PASSWORD --databases DATABASE1 DATABASE2 DATABASE3 > DATABASE-1-2-3-BACKUP-DATE.sql

Or if you just want to back up ALL the databases at one, use the “–all-databases” parameter to backup all the databases in the server in one easy step.

mysqldump --all-databases> ALL-DATABASES-DATE.sql

How to back up only the data structure.

While in development, most programmers only need to back up their database structure, not their test data. To do this, use the “–no-data” parameter.

mysqldump --no-data --databases DATABASE1 DATABASE2 DATABASE3 > DATABASE-1-2-3-Structure-Bakcup-DATE.sql

If you have large databases and not enough hard drive space to store them in the regular SQL, text format, you can compressing the backup file on the fly.

mysqldump --all-databases | bzip2 -c >all-databases-backup-date.sql.bz2
 
mysqldump --all-databases | gzip >all-databases-backup-date.sql.gz

If you are away from your servers, or have multiple servers that you manage, you can still backup your databases while you are away. Use the bash script below in conjunction with a cron-tab to automate your backups.

#!/bin/sh
date=`date -I`
mysqldump --all-databases | gzip > /var/mysql_backup/mysql-backup-all-$date.sql.gz

How to restore your databases.

If the unforeseen happens and you need to totally rebuild your databases, you can do so easily from the mysqldump SQL files.

mysql -u root -p PASSWORD DATABASE < DATABASE-BACKUP-DATE.sql

Mysql Synopsis:

mysql -u [username] -p [password] [database_to_restore] < [backup_file.sql]

Restoring from the compressed files.

gunzip < DATABASE-BACKUP.sql.gz | mysql -u root -p PASSWORD DATABASE

Or do 2 at a time.

cat DATABASE-BACKUP-1.sql DATABASE-BACKUP-2.sql | mysql -u root -p PASSWORD

Apache is the most widely used website hosting applications. At the time of this article, Apache is reported to have about 70% of the market. While Microsoft IIS has dropped dramatically to 30%. I’ve personally been using Apache for the last 15 years had have never ever had any problems with the application. All the problems I’ve had were related to typos in my configuration files.

To get Apache up and running, you need to do 2 things.

1) install apache
2) start apache

That is basically all you need to know, to get apache up and running. Fairly simple. But of course human nature creeps in and we want to start customizing and personalizing.

To Installing apache2 via apt-get:

sudo apt-get install apache2

Starting apache2 for the first time

sudo /etc/init.d/apache2 start

If you get this error:

apache2: Could not determine the server’s fully qualified domain name, using 127.0.0.1 for ServerName

You need to add “ServerName localhost” to the “/etc/apache2/conf.d/fqdn” file. This can all be done in a single command:

echo "ServerName localhost" | sudo tee /etc/apache2/conf.d/fqdn

If you need to Start|Stop|Reload|etc Apache’s httpd daemon on Ubuntu, you’d want to use the “/etc/init.d/apache2″ commands:

Usage: /etc/init.d/apache2 {
start
stop
restart
reload
force-reload
start-htcacheclean
stop-htcacheclean
status
}

If you are using any other operating system, you’d want to use the apachectl command:

apachectl – Apache HTTP Server Control Interface

/usr/sbin/apachectl

apachectl commands

start : Loads the org.apache.httpd launchd job.
stop, graceful-stop : Unloads the org.apache.httpd launchd job.
restart, graceful : Unloads, then loads the org.apache.httpd launchd job.
fullstatus : Displays  a  full status report from mod_status. For this to work, 
you need to have mod_status enabled on your server and a text-based 
browser such as lynx available on your system. The URL used to access the 
status report can be set by editing the STATUSURL variable in the script.
 
status : Displays a brief status report. Similar to the fullstatus option, except 
that the list of requests currently being served is omitted.
 
configtest : Run a configuration file syntax test. It parses the configuration files 
and either reports Syntax Ok or detailed information about  the  particular 
syntax error. This is equivalent to apachectl -t.
 
The following option was available in earlier versions but has been removed. 
(It is now built into the httpd daemon).
 
startssl : To  start  httpd  with  SSL  support,  you should edit your configuration 
file to include the relevant directives and then use the normal apachectl start.

The main files/directories you will be working with first are:
apache2.conf : the configuration file
sites-available : a directory to let Apache know what sites are available to be accessed by httpd
sites-enables : a directory to let Apache know which websites need to be loading on start

A list of the default apache2 configuration file structure in “/etc/apache2/”:

dmccoy@server:/etc/apache2# ls -l /etc/apache2/
total 48
-rw-r--r-- 1 root root 10104 2009-04-01 11:01 apache2.conf
drwxr-xr-x 2 root root  4096 2009-07-17 05:16 conf.d
-rw-r--r-- 1 root root   378 2009-04-01 11:01 envvars
-rw-r--r-- 1 root root     0 2009-07-13 10:57 httpd.conf
drwxr-xr-x 2 root root 12288 2009-07-17 05:16 mods-available
drwxr-xr-x 2 root root  4096 2009-07-13 10:57 mods-enabled
-rw-r--r-- 1 root root   513 2009-04-01 11:01 ports.conf
drwxr-xr-x 2 root root  4096 2009-07-22 00:54 sites-available
drwxr-xr-x 2 root root  4096 2009-07-13 10:57 sites-enabled

The default web site is located at:

/var/www/index.html

if you launch the domain in a web browser “http://localhost” you should see a message “It Works!”. If you are using the server version with out a gui, you can use “lynx http://localhost”. If lynx isn’t installed, you can use apt-get to install it.

If you need to maintain more then (1) website, you can use a Virtual Host to manage its location and access. Below is the default Virtual Host sample, located in “/etc/apache2/sites-available/default/”

There are (4) lines that you need to change.
1) You’ll need to change the ServerAdmin from webmaster@localhost to your_admin_account@domain.name email address.
2) If you are changing your default location of DocumentRoot /var/www, you’ll need to edit the path here. An instance where you’d need to change this directory, is if it was located on a different hard drive or hard drive partition.
3) If you change your DocumentRoot, then you’ll need to change the Directory from /var/www/ to your new path.
4) You’ll also need to change your log names to allow easier management of multiple domains. ErrorLog /var/log/apache2/error.log and CustomLog /var/log/apache2/access.log combined could be changed to domain.name-error.log and domain.name-access.log.

< VirtualHost *:80 >
        ServerAdmin webmaster@localhost
 
        DocumentRoot /var/www
        < Directory />
                Options FollowSymLinks
                AllowOverride None
        < /Directory >
        < Directory /var/www/ >
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        < /Directory >
 
        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        < Directory "/usr/lib/cgi-bin" >
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        < /Directory >
 
        ErrorLog /var/log/apache2/error.log
 
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
 
        CustomLog /var/log/apache2/access.log combined
 
    Alias /doc/ "/usr/share/doc/"
    < Directory "/usr/share/doc/" >
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    < /Directory >
< /VirtualHost >

Below is a sample VirtualHost file for a domain.

< VirtualHost *:80 >
        ServerAdmin dmccoy @ server .com
        ServerName server.com
        ServerAlias server
        DocumentRoot /var/www
        < Directory />
                Options FollowSymLinks
                AllowOverride None
        < /Directory >
        < Directory /var/www/server.com >
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        < /Directory >
 
        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        < Directory "/usr/lib/cgi-bin" >
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        < /Directory >
 
        ErrorLog /var/log/apache2/server.com-error.log
 
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
 
        CustomLog /var/log/apache2/server.com-access.log combined
 
    Alias /doc/ "/usr/share/doc/"
    < Directory "/usr/share/doc/" >
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    < /Directory >
< /VirtualHost >

You’ll Notice that I added (2) extra lines. I added “ServerName” and “ServerAlias“.

ServerName : Description : Hostname and port that the server uses to identify itself
ServerName : Syntax : ServerName fully-qualified-domain-name[:port]
ServerName : Usage : ServerName www.example.com:80

ServerAlias : Description : Alternate names for a host used when matching requests to name-virtual hosts
ServerAlias : Syntax : ServerAlias hostname [hostname] …
ServerAlias : Usage : ServerAlias server server2.domain.com server2

If by change you need manage another domain, you can copy the default virtualhost file and edit those same 4 lines to create another domain. And if you have another, you repeat the steps until all your domains are entered.

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/domain.name

Now that you’ve made your sites available to httpd (Apache), you can now enable them to be used by httpd. The easiest way to do this, is to make a symbolic link from the sites-available directory to the sites-enabled directory. Below is how I would enable the default domain for this server.

sudo ln -s /etc/apache2/sites-available/default /etc/apache2/sites-enabled/default

Once you have enabled all the domains you need, you can now restart Apache to make them active.

sudo /etc/init.d/apache2 restart
[sudo] password for dmccoy: 
 * Restarting web server apache2
 ... waiting    ...done.

For most server administrators doing system updates or installing patches via a command shell is normal, but for the average user trying to patch his new system, this task might be a little difficult. Opening a terminal and downloading a file, just by typing in a few words, is not as simple as clicking a download icon on a website, tho, once you learn how to use Wget or cURL, it will be.

From my understanding, Wget is a stand-alone application, that is command-line only. And, cURL is a cross-platfrom API library called libcurl. Personally, I like Wget better then cURL, but that’s just because I am bias.

If you want to know all the nitty-gritty differences, check out curl vs Wget.

Wget – The non-interactive network downloader. (I refer to it as, WWW Get or Web Get).

wget [option]… [URL]…

curl – transfer a URL

curl [options] [URL...]

How to download a file with wget:

wget http://domain.com/path/to/file.zip

How to download a file with cURL:

curl -O http://domain.com/path/to/file.zip
 
or
 
curl http://domain.com/path/to/file.zip > file.zip

Dstat is one of the best tools you can use for monitoring your server during peak times. It is a combination of vmstat, iostat, netstat, nfsstat and ifstat. This program allows you to access and view key elements of your system’s resources instantly.

dstat – versatile tool for generating system resource statistics

dstat [-afv] [options..] [delay [count]]

Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and 
system counters:
 
       dstat -dnyc -N eth0 -C total -f 5
       Checking dstat’s behaviour and the system’s impact on dstat:
 
       dstat -taf --debug
       Using the time plugin together with cpu, net, disk, system, load, proc and 
       topcpu plugins:
 
       dstat -tcndylp -M topcpu
       this is identical to
 
       dstat -M time,cpu,net,disk,sys,load,proc,topcpu
       Using dstat to relate cpu stats with interrupts per device:
 
       dstat -tcyif

Installing Dstat

sudo apt-get install dstat
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  dstat
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 58.0kB of archives.
After this operation, 406kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com jaunty/universe dstat 0.6.8-1 [58.0kB]
Fetched 58.0kB in 1s (50.0kB/s)
Selecting previously deselected package dstat.
(Reading database ... 24720 files and directories currently installed.)
Unpacking dstat (from .../archives/dstat_0.6.8-1_all.deb) ...
Processing triggers for man-db ...
Setting up dstat (0.6.8-1) ...

Dstat updates every second to display its information. Use “control C” to quit the application.

$ dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0   0|  13k   29k|   0     0 |   0     0 |   7    12 
  0   1  99   0   0   0|  32k    0 | 418B  404B|   0     0 |  11    14 
  0   0 100   0   0   0|   0     0 |1360B  420B|   0     0 |  17    14 
  0   0  99   1   0   0|   0   856k| 132B  420B|   0     0 |  22    22 
  0   0 100   0   0   0|   0     0 | 132B  420B|   0     0 |  10    14 
  0   0 100   0   0   0|   0     0 | 192B  420B|   0     0 |  12    16 ^C

A screen shot of the unmodified output of dstat:

Family Commands for dstat:

    Performance tools
       ifstat(1), iftop(8), iostat(1), mpstat(1), netstat(1), nfsstat(1), nstat, vmstat(1), xosview(1)
 
   Debugging tools
       htop(1), lslk(1), lsof(8), top(1)
 
   Process tracing
       ltrace(1), pmap(1), ps(1), pstack(1), strace(1)
 
   Binary debugging
       ldd(1), file(1), nm(1), objdump(1), readelf(1)
 
   Memory usage tools
       free(1), memusage, memusagestat, slabtop(1)
 
   Accounting tools
       dump-acct, dump-utmp, sa(8)
 
   Hardware debugging tools
       dmidecode, ifinfo(1), lsdev(1), lshal(1), lshw(1), lsmod(8), lspci(8), lsusb(8), smartctl(8), x86info(1)
 
   Application debugging
       mailstats(8), qshape(1)
 
   Xorg related tools
       xdpyinfo(1), xrestop(1)
 
   Other useful info
       proc(5)