Dave’s Ramblings

Linux / Unix27 Jun 2008 01:26 pm

This is a linux command line reference for common operations.

Command Description
• apropos whatis Show commands pertinent to string. See also threadsafe
• man -t man | ps2pdf - > man.pdf make a pdf of a manual page
which command Show full path name of command
time command See how long a command takes
• time cat Start stopwatch. Ctrl-d to stop. See also sw
• nice info Run a low priority command (The “info” reader in this case)
• renice 19 -p $$ Make shell (script) low priority. Use for non interactive tasks
dir navigation
• cd - Go to previous directory
• cd Go to $HOME directory
(cd dir && command) Go to dir, execute command and return to current dir
• pushd . Put current dir on stack so you can popd back to it
file searching
• alias l=’ls -l –color=auto’ quick dir listing
• ls -lrt List files by date. See also newest and find_mm_yyyy
• ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal
find -name ‘*.[ch]’ | xargs grep -E ‘expr’ Search ‘expr’ in this dir and below. See also findrepo
find -type f -print0 | xargs -r0 grep -F ‘example’ Search all regular files for ‘example’ in this dir and below
find -maxdepth 1 -type f | xargs grep -F ‘example’ Search all regular files for ‘example’ in this dir
find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done Process each item with multiple commands (in while loop)
• find -type f ! -perm -444 Find files not readable by all (useful for web site)
• find -type d ! -perm -111 Find dirs not accessible by all (useful for web site)
• locate -r ‘file[^/]*\.txt’ Search cached index for names. This re is like glob *file*.txt
• look reference Quickly search (sorted) dictionary for prefix
• grep –color reference /usr/share/dict/words Highlight occurances of regular expression in dictionary
archives and compression
gpg -c file Encrypt file
gpg file.gpg Decrypt file
tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir/
bzip2 -dc dir.tar.bz2 | tar -x Extract archive (use gzip instead of bzip2 for tar.gz files)
tar -c dir/ | gzip | gpg -c | ssh user@remote ‘dd of=dir.tar.gz.gpg’ Make encrypted archive of dir/ on remote machine
find dir/ -name ‘*.txt’ | tar -c –files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir/ and below
find dir/ -name ‘*.txt’ | xargs cp -a –target-directory=dir_txt/ –parents Make copy of subset of dir/ and below
( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir
( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/
( tar -c /dir/to/copy ) | ssh -C user@remote ‘cd /where/to/ && tar -x -p’ Copy (with permissions) copy/ dir to remote:/where/to/ dir
dd bs=1M if=/dev/sda | gzip | ssh user@remote ‘dd of=sda.gz’ Backup harddisk to remote machine
rsync (Network efficient file copier: Use the –dry-run option for testing)
rsync -P rsync://rsync.server.com/path/to/file file Only get diffs. Do multiple times for troublesome downloads
rsync –bwlimit=1000 fromfile tofile Locally copy with rate limit. It’s like nice for I/O
rsync -az -e ssh –delete ~/public_html/ remote.com:’~/public_html’ Mirror web site (using compression and encryption)
rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ Synchronize current directory with remote one
ssh (Secure SHell)
ssh $USER@$HOST command Run command on $HOST as $USER (default command=shell)
• ssh -f -Y $USER@$HOSTNAME xeyes Run GUI command on $HOSTNAME as $USER
scp -p -r $USER@$HOST: file dir/ Copy with permissions to $USER’s home directory on $HOST
ssh -g -L 8080:localhost:80 root@$HOST Forward connections to $HOSTNAME:8080 out to $HOST:80
ssh -R 1434:imap:143 root@$HOST Forward connections from $HOST:1434 in to imap:143
wget (multi purpose download tool)
• (cd cli && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html) Store local browsable version of a page to the current dir
wget -c http://www.example.com/large.file Continue downloading a partially downloaded file
wget -r -nd -np -l1 -A ‘*.jpg’ http://www.example.com/dir/ Download a set of files to the current directory
wget ftp://remote/file[1-9].iso/ FTP supports globbing directly
• wget -q -O- http://www.pixelbeat.org/timeline.html | grep ‘a href’ | head Process output directly
echo ‘wget url’ | at 01:00 Download url at 1AM to current dir
wget –limit-rate=20k url Do a low priority download (limit to 20KB/s in this case)
wget -nv –spider –force-html -i bookmarks.html Check links in a file
wget –mirror http://www.example.com/ Efficiently update a local copy of a site (handy from cron)
networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)
ethtool eth0 Show status of ethernet interface eth0
ethtool –change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed
iwconfig eth1 Show status of wireless interface eth1
iwconfig eth1 rate 1Mb/s fixed Manually set wireless interface speed
• iwlist scan List wireless networks in range
• ip link show List network interfaces
ip link set dev eth0 name wan Rename interface eth0 to wan
ip link set dev eth0 up Bring interface eth0 up (or down)
• ip addr show List addresses for interfaces
ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0)
• ip route show List routing table
ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254
• tc qdisc add dev lo root handle 1:0 netem delay 20msec Add 20ms latency to loopback device (for testing)
• tc qdisc del dev lo root Remove latency added above
• host pixelbeat.org Lookup DNS ip address for name or vice versa
• hostname -i Lookup local ip address (equivalent to host `hostname`)
• whois pixelbeat.org Lookup whois info for hostname or ip address
• netstat -tupl List internet services on a system
• netstat -tup List active connections to/from system
windows networking (Note samba is the package that provides all this windows specific networking support)
• smbtree Find windows machines. See also findsmb
nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address
smbclient -L windows_box List shares on windows machine or samba server
mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share
echo ‘message’ | smbclient -M windows_box Send popup to windows machine (off by default in XP sp2)
text manipulation (Note sed uses stdin and stdout, so if you want to edit files, append newfile)
sed ’s/string1/string2/g’ Replace string1 with string2
sed ’s/$.*$1/\12/g’ Modify anystring1 to anystring2
sed ‘/ *#/d; /^ *$/d’ Remove comments and blank lines
sed ‘:a; /\\$/N; s/\\\n//; ta’ Concatenate lines with trailing \
sed ’s/[ \t]*$//’ Remove trailing spaces from lines
sed ’s/$[\\`\\”$\\\\]$/\\\1/g’ Escape shell metacharacters active within double quotes
• seq 10 | sed “s/^/ /; s/ *$.\{7,\}$/\1/” Right align numbers
sed -n ‘1000p;1000q’ Print 1000th line
sed -n ‘10,20p;20q’ Print lines 10 to 20
sed -n ’s/.*$.*$< \/title>.*/\1/ip;T;q’ Extract title from HTML web page sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses • echo ‘Test’ | tr ‘[:lower:]’ ‘[:upper:]’ Case conversion • tr -dc ‘[:print:]’ < /dev/urandom Filter non printable characters • history | wc -l Count lines set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file) sort file1 file2 | uniq Union of unsorted files sort file1 file2 | uniq -d Intersection of unsorted files sort file1 file1 file2 | uniq -u Difference of unsorted files sort file1 file2 | uniq -u Symmetric Difference of unsorted files join -a1 -a2 file1 file2 Union of sorted files join file1 file2 Intersection of sorted files join -v2 file1 file2 Difference of sorted files join -v1 -v2 file1 file2 Symmetric Difference of sorted files math • echo ‘(1 + sqrt(5))/2′ | bc -l Quick math (Calculate φ). See also bc • echo ‘pad=20; min=64; (100*10^6)/((pad+min)*8)’ | bc More complex (int) e.g. This shows max FastE packet rate • echo ‘pad=20; min=64; print (100E6)/((pad+min)*8)’ | python Python handles scientific notation • echo ‘pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)’ | gnuplot -persist Plot FastE packet rate vs packet size • echo ‘obase=16; ibase=10; 64206′ | bc Base conversion (decimal to hexadecimal) • echo $((0×2dec)) Base conversion (hex to dec) ((shell arithmetic expansion)) • units -t ‘100m/9.72s’ ‘miles/hour’ Unit conversion (metric to imperial) • units -t ‘500GB’ ‘GiB’ Unit conversion (SI to IEC prefixes) • units -t ‘1 googol’ Definition lookup • seq 100 | (tr ‘\n’ +; echo 0) | bc Add a column of numbers. See also add and funcpy calendar • cal -3 Display a calendar • cal 9 1752 Display a calendar for a particular month year • date -d fri What date is it this friday. See also day • date –date=’25 Dec’ +%A What day does xmas fall on, this year • date –date=’@2147483647′ Convert seconds since the epoch (1970-01-01 UTC) to date • TZ=’:America/Los_Angeles’ date What time is it on West coast of US (use tzselect to find TZ) echo “mail -s ‘get the train’ P@draigBrady.com < /dev/null" | at 17:45 Email reminder • echo “DISPLAY=$DISPLAY xmessage cooker” | at “NOW + 30 minutes” Popup reminder locales • printf “%’d\n” 1234 Print number with thousands grouping appropriate to locale • BLOCK_SIZE=\’1 ls -l get ls to do thousands grouping appropriate to locale • echo “I live in `locale territory`” Extract info from locale database • LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes • locale | cut -d= -f1 | xargs locale -kc | less List fields available in locale database recode (Obsoletes iconv, dos2unix, unix2dos) • recode -l | less Show available conversions (aliases on each line) recode windows-1252.. file_to_change.txt Windows “ansi” to local charset (auto does CRLF conversion) recode utf-8/CRLF.. file_to_change.txt Windows utf8 to local charset recode iso-8859-15..utf8 file_to_change.txt Latin9 (western europe) to utf8 recode ../b64 < file.txt > file.b64 Base64 encode recode /qp.. < file.txt > file.qp Quoted printable decode recode ..HTML < file.txt > file.html Text to HTML • recode -lf windows-1252 | grep euro Lookup table of characters • echo -n 0×80 | recode latin-9/x1..dump Show what a code represents in latin-9 charmap • echo -n 0×20AC | recode ucs-2/x2..latin-9/x Show latin-9 encoding • echo -n 0×20AC | recode ucs-2/x2..utf-8/x Show utf-8 encoding CDs gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only) cdrecord -v dev=/dev/cdrom blank=fast Clear a CDRW gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom - Burn cdrom image (use dev=ATAPI -scanbus to confirm dev) cdparanoia -B Rip audio tracks from CD to wav files in current dir cdrecord -v dev=/dev/cdrom -audio *.wav Make audio CD from all wavs in current dir (see also cdrdao) oggenc –tracknum=’track’ track.cdda.wav -o ‘track.ogg’ Make ogg file from wav file disk space (See also FSlint) • ls -lSr Show files by size, biggest last • du -s * | sort -k1,1rn | head Show top disk users in current dir. See also dutop • df -h Show free space on mounted filesystems • df -i Show free inodes on mounted filesystems • fdisk -l Show disks partitions sizes and types (run as root) • rpm -q -a –qf ‘%10{SIZE}\t%{NAME}\n’ | sort -k1,1n List all packages by installed size (Bytes) on rpm distros • dpkg-query -W -f=’${Installed-Size;10}\t${Package}\n’ | sort -k1,1n List all packages by installed size (KBytes) on deb distros • dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space). See also truncate monitoring/debugging • tail -f /var/log/messages Monitor messages in a log file • strace -c ls >/dev/null Summarise/profile system calls made by command • strace -f -e open ls >/dev/null List system calls made by command • ltrace -f -e getenv ls >/dev/null List library calls made by command • lsof -p $$ List paths that process id has open • lsof ~ List processes that have specified path open • tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me • ps -e -o pid,args –forest List processes in a hierarchy • ps -e -o pcpu,cpu,nice,state,cputime,args –sort pcpu | sed ‘/^ 0.0 /d’ List processes by % cpu usage • ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem usage. See also ps_mem.py • ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process • ps -p 1,2 List info for particular process IDs • last reboot Show system reboot history • free -m Show amount of (remaining) RAM (-m displays in MB) • watch -n.1 ‘cat /proc/interrupts’ Watch changeable data continuously system information (see also sysinfo) (’#’ means root access is required) • uname -a Show kernel version and system architecture • head -n1 /etc/issue Show name and version of distribution • cat /proc/partitions Show all partitions registered on the system • grep MemTotal /proc/meminfo Show RAM total seen by the system • grep “model name” /proc/cpuinfo Show CPU(s) info • lspci -tv Show PCI info • lsusb -tv Show USB info • mount | column -t List mounted filesystems on the system (and align output) # dmidecode -q | less Display SMBIOS/DMI information # smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total # hdparm -i /dev/sda Show info about disk sda # hdparm -tT /dev/sda Do a read speed test on disk sda # badblocks -s /dev/sda Test for unreadable blocks on disk sda interactive (see also linux keyboard shortcuts) • readline Line editor used by bash, python, bc, gnuplot, … • screen Virtual terminals with detach capability, … • mc Powerful file manager that can browse rpm, tar, ftp, ssh, … • gnuplot Interactive/scriptable graphing • links Web browser • xdg-open http://www.pixelbeat.org/ open a file or url with the registered desktop application miscellaneous • alias hd=’od -Ax -tx1z -v’ Handy hexdump. (usage e.g.: • hd /proc/self/cmdline | less) • alias realpath=’readlink -f’ Canonicalize path. (usage e.g.: • realpath ~/../$USER) • set | grep $USER Search current environment touch -c -t 0304050607 file Set file timestamp (YYMMDDhhmm) • python -c “import SimpleHTTPServer as ws; ws.test()” Serve current directory tree at http://$HOSTNAME:8000/

Linux / Unix26 Jun 2008 03:17 pm

Keyboard shortcuts for BASH

Nothing to earth shaking here, below are all the keyboard shortcuts for the BASH shell that I can think of.. I know there are more and you can customize them to fit.

One note, if you habitually use ’screen’ Ctrl-a will not move the cursor to the line start, instead it follows the default keymapping for screen and will await the command to change screens.

Moving around

Ctrl-b Move the cursor one character ⇦ to the left
Ctrl-f Move the cursor one character ⇨ to the right
Alt-b Move the cursor one word ⇦ to the left
Alt-f Move the cursor one word ⇨ to the right
Ctrl-a Move the cursor ⇤ to the start of the line
Ctrl-e Move the cursor ⇥ to the end of the line
Ctrl-x-x Move the cursor ⇤⇥ to the start, and to the end again

Cut, copy and paste

Backspace Delete the character ⇦ to the left of the cursor
DEL/Ctrl-d Delete the character underneath the cursor
Ctrl-u Delete everything ⇤ from the cursor back to the line start
Ctrl-k Delete everything ⇥ from the cursor to the end of the line
Alt-d Delete word ⇨ untill before the next word boundary
Ctrl –w Delete word ⇦ untill after the previous word boundary
Ctrl-y Yank/Paste prev. killed text at the cursor position
Alt-y Yank/Paste prev. prev. killed text at the cursor position

History
Ctrl-p Move in history one line up before this line
Ctrl-n Move in history one line down after this line
A-> Move in history all the lines down to the line being eneterd
Ctrl-r Incrementally search the line in history backward
Ctrl-s Incrementally search the line in history forward
Ctrl-j End incremental search
Ctrl-g Abort incremental search AND restore the original line
Alt-Ctrl-y Yank/Paste arg.1 of prev. command at the cursor position
Alt-. / Alt-, Yank/Paste last arg. Of prev. command at the cursor position

Undo
Ctrl-_/Ctrl-x/Ctrl-u Undo the last editing command
Alt-r undo all changes made to this line
Ctrl-l Clear the screen

Completion
TAB Auto-complete a name
Alt-/ Auto-complete a name (without smart completion)
Alt-? /TAB-TAB List the possible completion of the preceding text
Alt-* Insert all possible completions of the preceding text

Transpose (Twiddle)
Ctrl-t Transpose char. Before cursor with the character at the cursor
Alt-t Transpose word before the cursor with the word at the cursor

Ctrl-d Exit the current shell
Ctrl-z Puts whatever you are running into a suspended background process.

Linux / Unix24 Jun 2008 01:41 pm

Find and change file owners & groups

Consolidating users or changing owners for files scattered across file systems can be a royal pain for Unix/Linux, For Winblows; I don't even want to try and imagine the quantity of Excedrin it might take to accomplish this task (but who cares about Winblows?)

The simple solution is keyed in just as it sounds... find all the files/directories owned by user1 and chown them to user2, we'll also say that user2 is in a new group called group2.

Sure you can do a LOT of keyboard surfing and get it accomplished, how about two short lines and let the server take care of it for you....

First you'lll need the UID (and GID) for user1 (hint look in /etc/passwd and /etc/group) and use the following commands;

PLAIN TEXT

CODE:

find . -uid 5015 -exec chown user2 {} \;
find . -gid 5015 -exec chgrp group2 {} \;

You'd be hard pressed to accomplish this much faster and without a lot of effort.

Happy Hacking

Linux / Unix16 May 2008 10:29 am

How to enable .htaccess .htpasswd for Apache

How to enable .htaccess on .htpasswd for Apache.

Here are some short steps about how to password protect websites (or certain directories) using on an Apache webserver.

Note: this assumes you already have Apache installed and running correctly. This writeup is based on Slackware 12.0 & Apache 2.2.8 however the instructions should apply to any previous version of Apache or Linux/Unix builds. YMMV

First enabling .htaccess is simple. Open your active httpd.conf (mine is located @ /etc/httpd/conf/httpd.conf) in your favorite editor and look for the following lines

# First, we configure the “default” to be a very restrictive set of
# features.
#
Options FollowSymLinks
AllowOverride None

Change AllowOverride to All:
Options FollowSymLinks
AllowOverride All

Next, look for:

#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be “All”, “None”, or any combination of the keywords:
# Options FileInfo AuthConfig Limit
#
AllowOverride None

Change this to:

AllowOverride All

Restart apache:

[root@server]# /usr/bin/apachectl restart

As simple as that .htaccess is now enabled for your server.

Now lets enable it for the directory/site you wish to protect.

Shell in and navigate to the web directory that you wish to protect

[rss@server]$ cd public_html/protected
[rss@server protected]$

Find out your directory path:

[rss@server protected]$ pwd
/home/rss/public_html/protected

Create the .htpasswd file

[rss@server protected]$ htpasswd -mc .htpasswd noob
New password:
Re-type new password:
Adding password for user noob
[rss@server protected]$

Create an .htaccess file

[rss@server protected]$ touch .htaccess

Add the following lines to .htaccess using your favorite text editor
Note: You must change the bolded entries to your own settings

AuthType Basic
AuthUserFile /home/rss/public_html/protected/.htpasswd
AuthGroupFile /dev/null
AuthName “Protected Area”
require valid-user

Save the file and exit to console.

Check permissions
Note: Make sure the permissions are set correctly on the .htaccess and .htpasswd files

[rss@server protected]$ ls -al .ht*
-rw-r–r– 1 rss public 129 Apr 30 00:19 .htaccess
-rw-r–r– 1 rss public 19 Apr 30 00:23 .htpasswd
[rss@server protected]$

If for some reason the permissions are not set correctly, chmod them (644)

[rss@server protected]$ chmod 644 .ht*

Add more users to the password file
Note: If you want to add more users to access the directory, use the htpasswd command:

[rss@server protected]$ htpasswd -m .htpasswd newuser
New password:
Re-type new password:
Adding password for user newuser

That's really all there is to it.. I would recommend not storing the .htpasswd file in the directory that it's protecting (or even in a directory that is being served). Move the .htpasswd file to another location and change the AuthUserFile line within the .htaccess file to match the new location.

Linux / Unix06 May 2008 03:49 pm

Listing installed Perl Modules

Simple one line command for listing all installed Perl modules

PLAIN TEXT

CODE: