Difference: UnixNotes (1 vs. 51)

I had a problem where xdg-open (and all of GNOME) was using Handbrake to open video/mp4 files when I had configured GNOME (using the settings app) and xdg-open (using xdg-mime) to use VLC. It turns out that xdg-open delegates to gvfs-open in a GNOME environment, and for some reason gvfs-open didn't respect what I had done in the settings app. gvfs-mime is supposed to allow you to set the preferences, but they didn't "stick". In the end I had to strace gvfs_mime to see what files it was reading, and then hand-edit ~/.local/share/applications/mimeapps.list. There's a section called [Added Associations] that seems to have precedence over [Default Applications]. When I removed ghb from the added associations gvfs-mime started being willing to use vlc to open videos.

tcpdump

Sometimes there's no substitute for looking at what's crossing the wire. Assuming that it's not encrypted, of course.

$ sudo tcpdump -s 0 -vv -A "host 173.76.76.13 and tcp port http" | tee capture.txt

That command dumps all HTTP traffic to 173.76.76.13 into capture.txt and displays it on the console. -s 0 means no packet size limit, -A means dump in ASCII (good for HTTP), and -vv means verbose.

HOWTO enable smartd: http://www.cyberciti.biz/tips/monitoring-hard-disk-health-with-smartd-under-linux-or-unix-operating-systems.html

If you find that the disk reports any Current_Pending_Sector errors then this will help: http://smartmontools.sourceforge.net/badblockhowto.html

http://noone.org/blog/English/Computer/Shell/How%20to%20find%20broken%20symlinks.html - How to find broken symlinks

$ find -L . -type l

When I moved from one cloud machine to another I had to decide what to sync and what to leave behind. I was also moving from Debian to Ubuntu so I couldn't just sync everything.

What I took:

source                target
----------------------------
/home                 /
/var/mail             /var
/var/lib/squirrelmail /var/lib

rsync -rtPv --rsh="ssh -p 22222" --only-write-batch=/usb-drive /source-dir/* login@target-host:/target-dir

sudo rsync --rsh "ssh -i user_private_key" --rsync-path "sudo rsync" user@host:'/remotepath' localpath

• PostfixNotes

Hardening Linux Machines

http://docstore.mik.ua/orelly/unix/sedawk/ch06_01.htm#SEDAWK-CH-6-SECT-1

e.g. "unfold" two lines back to a single line:

sed '/@RolesAllowed($/{                                                
N
s/\n//
}' src

Here's an article that talks about how to assign labels to partitions to make booting from USB devices more flexible: http://www.ibm.com/developerworks/linux/library/l-boot-rootfs/

open a connection from inside a firewall allowing tunnelled logins back through: ssh -Nf -R 22222:localhost:22 login@hostname

tunnel back up that tunnel and use a USB disk sneakernet to sync a big directory: rsync -rtPv --rsh="ssh -p 22222" --only-write-batch=/usb-drive /source-dir/* login@target-host:/target-dir

I had to swap a motherboard in my little home server when the old one went south, and when I booted the new one I got a message about udev: renamed network interface eth0 to eth1 which seemed kind of peculiar. It turns out that udev tries to make sure that a given network interface (identified by its MAC address) will always have the same name, even if you change the machine's hardware (e.g. add another network card). It uses a file called /etc/udev/rules.d/70-persistent-net.rules for this purpose. Since it already had some entries for eth0 and it hadn't seen the new motherboard's MAC before, it gave it the name "eth1". This is reasonable behavior once you figure it out. A quick edit to that file to change the name to "eth0" and I was all set.

  mknbi-linux --rootdir=rom --ip=dhcp arch/i386/boot/bzImage > /tmp/bzImage-2.6.4
  sudo cp /tmp/bzImage-2.6.4 /tftpboot/lts/
  make modules-install
  sudo mv /lib/modules/2.4.6 /usr/local/ltsp/dickless/lib/modules  

mknbi-linux --rootdir=rom --ip=dhcp arch/i386/boot/bzImage > /tmp/bzImage-2.6.4  sudo cp /tmp/bzImage-2.6.4 /tftpboot/lts/  make modules-install  sudo mv /lib/modules/2.4.6 /usr/local/ltsp/dickless/lib/modules  

HAL is responsible for mounting USB drives when they're plugged in. It will use the partition's label as a mount point if it can or it will fall back to something like "disk". http://www.debuntu.org/device-partition-labeling talks about how to set the labels.

Read from remote host www.caboteria.org: Connection reset by peer Connection to www.caboteria.org closed. 

Date: Thu, 22 Nov 2001 23:50:39 +0300 From: Diomidis Spinellis  Subject: Risks of the space character in Unix filenames                      The root of the problem reported in the "Glitch in iTunes Deletes Drives   (Solomon, RISKS-21.74)" article is the default way the Unix shell handles  filenames with embedded spaces.  Although a space can legally appear in a  Unix filename, such an occurrence is not usual; Unix filenames tend to be terse, often even shorter than a single word, (e.g. "src", "doc", "etc",    "bin") so they can be swiftly typed.  A number of more recent and supposedly user-friendly operating systems like the Microsoft Windows family, and, I    understand, the MacOS, use longer and more descriptive file names          ("Documents and Settings", "Program Files").  Many of these filenames contain spaces; the ones I listed are by default used by Windows 2000 as the location to store user data and application files (the equivalent of /home/username and /bin under Unix).  As Unix-style tools and relevant applications are increasingly ported to run under Windows (see for example [1, 2, 3] and my Windows outwit tool suite described in [4]) or natively run under Mac OS X, problems and associated    Risks arise.  The main reason is that some often-used Unix shell constructs fail when applied to filenames containing a space character.  Unfortunately, these constructs appear in many existing programs, and even in the writings of the original system developers, who, in all fairness, could not have     foreseen how their tools would have been used 25 years after their           conception.          Technically, the problem manifests itself when field splitting (the process by which the shell splits input into words) is naively applied on the output of an expansion that generates filenames with embedded spaces.  Consider the following example, appearing on page 95 of one of the classic texts on Unix programming [5]:           for i in ch2.*     echo $i:     diff -b old/$i $i     echo   done  The above code will compare all files matching the ch2.* pattern in the current directory with copies presumably stored in the directory called "old".  Consider what will happen when the code is applied to a file called "ch2.figure 3.dot" (notice the space between the word figure and the "3"). The shell variable i will be set to the correct filename, but then the shell will execute the "diff" command with the following argument list (customarily passed to C programs in the argv array):   argv[0] = "diff"   argv[1] = "-b"   argv[2] = "old/ch2.figure"   argv[3] = "3.dot"   argv[4] = "ch2.figure"   argv[5] = "3.dot"  and diff will complain   diff: extra operand as more than two filenames were passed as arguments.  This happens, because words are expanded by most Unix shells in the following order:   1. Parameter (including variable) expansion, command substitution.   2. Field splitting. As a result, the variable $i is first expanded into "ch2.figure 3" and then the result is split into fields for further processing or for passing them as arguments to a command.  The most common dangerous constructs that can appear in step 1 are variable references (e.g. $PATH, $word) and commands inside backquotes (e.g. `find . -type f -name 'ch2.*'`).  These dangerous constructs are quite common, appearing among other places in the original article describing the Bourne shell [6] (for i in * do if test -d $d/$i [...]), in other scripts in the reference of the original example [5 p. 141, 143], and even in quite recent work by the same authors [7, p. 149].  It is also prevalent in existing operating system tools; I counted 43 occurrences of one suspicious pattern ("$*") in a NetBSD source tree, 8 in a FreeBSD command path, and 49 in the shell scripts of a Mandrake Linux distribution.  The Unix world is definitely not ready to deal with filenames containing the space character.  Avoiding this problem is not trivial.  A radical solution would be to change the value of the shell's "internal field separator" (IFS) variable.  This variable contains the characters that shell uses to split words.  Its default value is ".  This solution however would break more things than it would fix, since most scripts expect words to be separated by spaces.  As an example the construct "A='ls -l';$A" would not work.  The most practical solution is to manually enclose variables inside double quotes when using them in contexts where only a single word is normally expected.  The shell will still expand the variable inside the quotes, but will treat the result as a single word.  Thus the offending part in the original example should have been written as:   diff -b "old/$i" "$i" In addition, whenever a shell script uses the variable $* to obtain the values of all parameters passed to a script, the $* variable should be replaced by the variable $@, again inside double quotes.  Thus the common code pattern   for arg in $* should be written as   for arg in "$@" Interestingly, Kernighan and Pike were aware of the $* problem and the above solution since 1984; they aptly characterize the "$@" solution as "almost black magic" [5 p. 161].  Still, these changes will not correctly handle filenames with embedded whitespace returned from a command substitution.  In this case, temporarily changing the IFS variable before executing a command may be the only feasible solution.  The following example illustrates this approach:   # Save original IFS   OFS="$IFS"   # Set IFS to newline   IFS='   '   # The find command might output filenames with spaces   wc -l `find . -type f`   # Restore original IFS   IFS="$OFS"  By searching existing shell scripts for the patterns I described and applying the suggested changes most problems can be solved.  Other scripting languages like Tcl and, to a lesser extend, Perl may also have problems dealing with filenames with spaces.  Similar approaches (appropriate quoting in Perl "eval" blocks and use of the "list" command in Tcl) can be used to avoid these problems.  References  [1] David G. Korn. Porting Unix to Windows NT. In Proceedings of the USENIX 1997 Annual Technical Conference, Anaheim, CA, USA, January 1997. Usenix Association. [2] Geoffrey J. Noer. Cygwin32: A free Win32 porting layer for UNIX applications. In Proceedings of the 2nd USENIX Windows NT Symposium, Seattle, WA, USA, August 1998. Usenix Association. [3] Stephen R. Walli. OPENNT: UNIX application portability to Windows NT via an alternative environment subsystem. In Proceedings of the USENIX Windows NT Symposium, Seattle, WA, USA, August 1997. Usenix Association. [4] Diomidis Spinellis. Outwit: Unix tool-based programming meets the Windows world. In USENIX 2000 Technical Conference Proceedings, pages 149-158, San Diego, CA, USA, June 2000. Usenix Association.  [5] Brian W. Kernighan and Rob Pike. The UNIX Programming Environment. Prentice-Hall, 1984. [6] S. R. Bourne. The UNIX shell. Bell System Technical Journal, 57(6):65-84 July/August 1978.  (Also appears in volume 2 of the Unix Programmer's Manual and in AT & T, UNIX System Readings and Applications, volume I. Prentice-Hall, 1987.) [7] Brian W. Kernighan and Rob Pike. The Practice of Programming. Addison-Wesley, 1999.  Diomidis Spinellis - http://www.dmst.aueb.gr/dds/ Athens University of Economics and Business (AUEB) 

mknbi-linux --rootdir=rom --ip=dhcp arch/i386/boot/bzImage > /tmp/bzImage-2.6.4 sudo cp /tmp/bzImage-2.6.4 /tftpboot/lts/ make modules-install sudo mv /lib/modules/2.4.6 /usr/local/ltsp/dickless/lib/modules 

I got a usb-attached disk drive to make backups to, and it spins the drive down after a while to save power. That's cool, but the problem is that Linux barfs when it tries to access the drive:

Apr  9 06:19:01 voom kernel: sd 0:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Apr  9 06:19:01 voom kernel:     Additional sense: Logical unit not ready, initializing command required
Apr  9 06:19:01 voom kernel: end_request: I/O error, dev sda, sector 12375

This might help: http://www.nslu2-linux.org/wiki/FAQ/DealWithAutoSpinDownOnSeagateFreeAgent

When I was logged into warthog and plugged Tory's ipod into its cradle, it would get mounted automagically. When Tory tried this it didn't work. It seems that the st of programs to manage things like ipods is pretty complicated, but there's some useful info at http://www.freedesktop.org/wiki/Software/HalFAQ and especially http://www.mythic-beasts.com/~mark/random/hal/.

In our case it turned out that Tory wasn't a member of the plugdev group.

rdiff-backup - http://www.nongnu.org/rdiff-backup/docs.html

Python program that uses rdiff to generate efficient differential backups. Straightforward command line. Seems to use a lot of CPU.

rsnapshot - http://www.rsnapshot.org/

Similar idea, written in perl. Seems to be more config-file oriented less command-line oriented.

http://www.mikerubel.org/computers/rsync_snapshots/
http://blog.interlinked.org/tutorials/rsync_time_machine.html

if you're like me you'll probably blow away the windows MBR in the process, so here's how to fix it: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/bootcons_fixmbr.mspx

site - http://www.gnu.org/software/grub/

manual - http://www.gnu.org/software/grub/manual/

burn a CD - http://www.gnu.org/software/grub/manual/html_node/Making-a-GRUB-bootable-CD_002dROM.html#Making-a-GRUB-bootable-CD_002dROM

a good way to scan the partitions is find /grub/stage1 since that's about the shortest filename you can reliably look for in a grub installation.

Two HOWTO's that I found useful for setting up chroot jails are: http://www.tjw.org/chroot-login-HOWTO/, http://kegel.com/crosstool/current/doc/chroot-login-howto.html

Build kernel, need CONFIG_IP_PNP=y, CONFIG_ROOT_NFS=y, and of course the driver for the ethernet card should be built-in, i.e. not a module.

mknbi-linux --rootdir=rom --ip=dhcp arch/i386/boot/bzImage > /tmp/bzImage-2.6.4
sudo cp /tmp/bzImage-2.6.4 /tftpboot/lts/
make modules-install
sudo mv /lib/modules/2.4.6 /usr/local/ltsp/dickless/lib/modules

• FedoraNotes

• PluggableAuthenticationModules

• PostfixComcastAol

See also http://syslinux.hackerdojo.com/pxe.php if you've got a motherboard with a PXE bios.

Info on diskless workstations: http://lists.debian.org/debian-devel/2001/debian-devel-200104/msg00647.html I wasn't able to get reliable performance using the user-space nfs server (many errors in apt-get) but it seems to be reliable using the kernel nfs server.

-- TobyCabot - 17 Oct 2001-25 Mar 2002

I like the Sawfish window manager, and my favorite theme is STPflat, by rowan@stasis.org. See http://stasis.org/~rowan/graphics.html.

-- TobyCabot - 17 Oct 2001-20 Mar 2002

• UnixPerformanceTips

• DebianTips
• MandrakeTips

Date: Thu, 22 Nov 2001 23:50:39 +0300
From: Diomidis Spinellis 
Subject: Risks of the space character in Unix filenames						  

The root of the problem reported in the "Glitch in iTunes Deletes Drives  
(Solomon, RISKS-21.74)" article is the default way the Unix shell handles 
filenames with embedded spaces.  Although a space can legally appear in a 
Unix filename, such an occurrence is not usual; Unix filenames tend to be
terse, often even shorter than a single word, (e.g. "src", "doc", "etc",	
"bin") so they can be swiftly typed.  A number of more recent and supposedly
user-friendly operating systems like the Microsoft Windows family, and, I	
understand, the MacOS, use longer and more descriptive file names			
("Documents and Settings", "Program Files").  Many of these filenames
contain spaces; the ones I listed are by default used by Windows 2000 as the
location to store user data and application files (the equivalent of
/home/username and /bin under Unix).

As Unix-style tools and relevant applications are increasingly ported to run
under Windows (see for example [1, 2, 3] and my Windows outwit tool suite
described in [4]) or natively run under Mac OS X, problems and associated	
Risks arise.  The main reason is that some often-used Unix shell constructs
fail when applied to filenames containing a space character.  Unfortunately,
these constructs appear in many existing programs, and even in the writings
of the original system developers, who, in all fairness, could not have	 
foreseen how their tools would have been used 25 years after their			 
conception.		  

Technically, the problem manifests itself when field splitting (the process
by which the shell splits input into words) is naively applied on the output
of an expansion that generates filenames with embedded spaces.  Consider the
following example, appearing on page 95 of one of the classic texts on Unix
programming [5]:		  
  for i in ch2.*
	 echo $i:
	 diff -b old/$i $i
	 echo
  done

The above code will compare all files matching the ch2.* pattern in the
current directory with copies presumably stored in the directory called
"old".  Consider what will happen when the code is applied to a file called
"ch2.figure 3.dot" (notice the space between the word figure and the "3").
The shell variable i will be set to the correct filename, but then the shell
will execute the "diff" command with the following argument list
(customarily passed to C programs in the argv array):
  argv[0] = "diff"
  argv[1] = "-b"
  argv[2] = "old/ch2.figure"
  argv[3] = "3.dot"
  argv[4] = "ch2.figure"
  argv[5] = "3.dot"

and diff will complain
  diff: extra operand
as more than two filenames were passed as arguments.  This happens,
because words are expanded by most Unix shells in the following order:
  1. Parameter (including variable) expansion, command substitution.
  2. Field splitting.
As a result, the variable $i is first expanded into "ch2.figure 3" and
then the result is split into fields for further processing or for
passing them as arguments to a command.

The most common dangerous constructs that can appear in step 1 are variable
references (e.g. $PATH, $word) and commands inside backquotes (e.g. `find
. -type f -name 'ch2.*'`).  These dangerous constructs are quite common,
appearing among other places in the original article describing the Bourne
shell [6] (for i in * do if test -d $d/$i [...]), in other scripts in the
reference of the original example [5 p. 141, 143], and even in quite recent
work by the same authors [7, p. 149].  It is also prevalent in existing
operating system tools; I counted 43 occurrences of one suspicious pattern
("$*") in a NetBSD source tree, 8 in a FreeBSD command path, and 49 in the
shell scripts of a Mandrake Linux distribution.  The Unix world is
definitely not ready to deal with filenames containing the space character.

Avoiding this problem is not trivial.  A radical solution would be to change
the value of the shell's "internal field separator" (IFS) variable.  This
variable contains the characters that shell uses to split words.  Its
default value is ".  This solution however would break
more things than it would fix, since most scripts expect words to be
separated by spaces.  As an example the construct "A='ls -l';$A" would not
work.  The most practical solution is to manually enclose variables inside
double quotes when using them in contexts where only a single word is
normally expected.  The shell will still expand the variable inside the
quotes, but will treat the result as a single word.  Thus the offending part
in the original example should have been written as:
  diff -b "old/$i" "$i"
In addition, whenever a shell script uses the variable $* to obtain the
values of all parameters passed to a script, the $* variable should be
replaced by the variable $@, again inside double quotes.  Thus the
common code pattern
  for arg in $*
should be written as
  for arg in "$@"
Interestingly, Kernighan and Pike were aware of the $* problem and the above
solution since 1984; they aptly characterize the "$@" solution as "almost
black magic" [5 p. 161].

Still, these changes will not correctly handle filenames with embedded
whitespace returned from a command substitution.  In this case, temporarily
changing the IFS variable before executing a command may be the only
feasible solution.  The following example illustrates this approach:
  # Save original IFS
  OFS="$IFS"
  # Set IFS to newline
  IFS='
  '
  # The find command might output filenames with spaces
  wc -l `find . -type f`
  # Restore original IFS
  IFS="$OFS"

By searching existing shell scripts for the patterns I described and
applying the suggested changes most problems can be solved.  Other scripting
languages like Tcl and, to a lesser extend, Perl may also have problems
dealing with filenames with spaces.  Similar approaches (appropriate quoting
in Perl "eval" blocks and use of the "list" command in Tcl) can be used to
avoid these problems.

References

[1] David G. Korn. Porting Unix to Windows NT. In Proceedings of the
USENIX 1997 Annual Technical Conference, Anaheim, CA, USA, January 1997.
Usenix Association.
[2] Geoffrey J. Noer. Cygwin32: A free Win32 porting layer for UNIX
applications. In Proceedings of the 2nd USENIX Windows NT Symposium,
Seattle, WA, USA, August 1998. Usenix Association.
[3] Stephen R. Walli. OPENNT: UNIX application portability to Windows NT
via an alternative environment subsystem. In Proceedings of the USENIX
Windows NT Symposium, Seattle, WA, USA, August 1997. Usenix Association.
[4] Diomidis Spinellis. Outwit: Unix tool-based programming meets the
Windows world. In USENIX 2000 Technical Conference Proceedings, pages
149-158, San Diego, CA, USA, June 2000. Usenix Association.

[5] Brian W. Kernighan and Rob Pike. The UNIX Programming Environment.
Prentice-Hall, 1984.
[6] S. R. Bourne. The UNIX shell. Bell System Technical Journal,
57(6):65-84 July/August 1978.  (Also appears in volume 2 of the Unix
Programmer's Manual and in AT & T, UNIX System Readings and
Applications, volume I. Prentice-Hall, 1987.)
[7] Brian W. Kernighan and Rob Pike. The Practice of Programming.
Addison-Wesley, 1999.

Diomidis Spinellis - http://www.dmst.aueb.gr/dds/
Athens University of Economics and Business (AUEB)

I had a problem once trying to ssh from work to my machine at home. Everything worked fine until I left the connection idle for a while. When I hit a key I got

Read from remote host www.caboteria.org: Connection reset by peer
Connection to www.caboteria.org closed.

Setting ssh's KeepAlive parameter on the client and server didn't seem to do much good. I found a patch that helped, though:

http://www.icl.isc.tohoku.ac.jp/~hgot/sources/openssh-watchdog.html

The "contributed" patch at the bottom is close to the version that ships with Debian Potato, and although it gives errors when you try to apply it it's pretty small and relatively easy to cut-n-paste by hand.

-- TobyCabot - 17 Oct 2001