Archive for the 'GNU/Linux' Category

12 DecScript to Convert Windows-1252 files to UTF-8

I had several hundred (over 1000) HTML files in a directory. They were unfortunately encoded in Windows-1252 and I wanted them all converted to UTF-8, but I was not willing to open the files one by one or feed their names to a script (there’s too many) so I needed a script that would operate on the whole directory and spit out the converted files in one fell swoop.

If you’re not familiar with encodings the visual problem one sees is that Firefox displays little black diamonds with question marks inside them for characters it doesn’t understand (I think they’re mostly tabs, spaces, and em-dashes in this case.)

With help from friends and the internet I learned about the GNU/Linux command-line tool iconv which handled this perfectly. Here’s the bash script I used that made it work on the entire directory at once:

#/bin/bash
LIST=`ls *.html`
for i in $LIST;
do iconv -f WINDOWS-1252 -t UTF8 $i -o $i.”utf8″;
mv $i.”utf8″ $i;
done

It seems that iconv requires a new name for the output file, so the above script temporarily names them *.utf and then moves them back over the original .html files. Hopefully this helps someone else.

15 AugDebian on the Openmoko Neo FreeRunner

It was inevitable. One can now run the entire Debian distribution (ARM port) on the Openmoko Neo Freerunner. Slashdot previously covered the July 4th launch of this GNU/Linux-based smartphone, which is open down to its core, with the company providing CAD files and schematics for the phone. Openmoko released an update to their software stack earlier this month, called Om2008.8, which is still a work in progress. But now one can use these instructions on the Debian wiki to open up the possibility of using apt-get to access Debian’s more than 20,000 applications–on your phone, which due to integration with freesmartphone.org efforts, can also actually be used as a phone. There were previously efforts to run Debian on the predecessor product to the Neo FreeRunner, the Neo 1973, but with the wider adoption of the Neo FreeRunner and the hard work of many Debian developers at the ongoing DebConf8, carrying Debian in your pocket has just gotten a lot easier.

25 MayHistory of SCO (funny)

I wrote an article that, in part, explained the SCO v. IBM lawsuit up to that point in time. That wouldn’t have been necessary had this history of SCO been around. Too funny.

07 AprMultiple WordPress blogs on a Debian server

This HOWTO assumes:

1. You have a server running at least Debian Etch (or are otherwise able to install the wordpress .deb).

2. Your server has Apache 2, MySQL, and PHP installed.

3. You want to use a single WordPress installation to host multiple blogs across several domains.

Ok, let’s get started.

Step 1: # sudo apt-get install wordpress

Handle a few pesky permissions problems with:

# sudo chown :www-data /etc/wordpress/*
# sudo mkdir /usr/share/wordpress/wp-content/uploads
# sudo chown www-data:www-data /usr/share/wordpress/wp-content/uploads
# sudo chown -R www-data:www-data /usr/share/wordpress/wp-content/themes

Step 2: Set up Apache 2 so that exampledomain.org points to /usr/share/wordpress —Here’s one way, and it also assumes you want www.exampledomain.org to redirect to exampledomain.org. You could modify this to have it redirect to blog.exampledomain.org if you want it to go there instead.


/etc/apache2/sites-available/exampledomain.org

<VirtualHost *:80>
ServerName www.exampledomain.org
DocumentRoot /var/www/www.exampledomain.org/
<Directory /var/www/www.exampledomain.org/>
AllowOverride All
Order Deny,Allow
Allow from all
</Directory>
</VirtualHost>

<VirtualHost *:80>
ServerName exampledomain.org
DocumentRoot /var/www/exampledomain.org/
<Directory /var/www/exampledomain.org/>
AllowOverride All
Order Deny,Allow
Allow from all
</Directory>
</VirtualHost>

# sudo a2ensite exampledomain.org
# sudo /etc/init.d/apache2 reload

# sudo ln -s /usr/share/wordpress /var/www/exampledomain.org

/var/www/www.exampledomain.org/.htaccess
Redirect / http://exampledomain.org

[Go online and confirm the redirect is working—don't worry that you get a WordPress error. It's not supposed to work yet.]

# cd /usr/share/doc/wordpress/examples
# sudo sh setup-mysql -n exampleuser exampledomain.com

Note that the username "exampleuser" must be no more than 16 characters (mysql limitation).

Step 3: Do the normal web-based WordPress install (I suggest going to Options|Miscellaneous and giving each blog a unique subdirectory for its uploads otherwise they’ll all be mixed together) and then repeat step 2 for each additional domain.

You’re done! A single Debian WordPress installation is now hosting multiple blogs!

07 AprHello world!

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

26 MayMacBook Triple-Boot HowTo (With Debian Etch and XP Upgrade version).

Triple-Boot on Brian's Black MacBook (Thumbnail) 0. Buy a MacBook. I suggest 120Gb HD and 1 or 2GB RAM. Buy an Upgrade version of Windows XP (so long as you have an older Windows CD)–My cost $99. Buy an external CD-Rom drive with USB connection–My cost $40. Find two blank CD-Rs (one for XP Drivers, one for Debian Etch Net Install). An additional CD-R or USB-key will come in handy for subsequent configuration.

1. Prepare XP Drivers. Follow the instructions at OnMac up to a point. That is:

Take the steps necessary to install Boot Camp and use that first blank CD-R to burn the Windows XP drivers. Then QUIT Boot Camp without using it to partition your drive!

2. Partition your hard drive. (Um, you have a back-up, right?)

Type: diskutil list
It should give you something like:

  /dev/disk0
   #:                   type name               size      identifier
   0:  GUID_partition_scheme                    *111.8 GB disk0
   1:                    EFI                    200.0 MB  disk0s1
   2:              Apple_HFS Macintosh HD       111.4 GB  disk0s2 

Find the identifier of your Mac OS X partion (in this case disk0s2) and type the following in order to repartition the drive. In the example below, I have repartitioned a 120Gb Hard disk to contain 60Gb OSX, 20Gb Linux and 31Gb windows partitions. You can change the volume names/sizes but not the order. (Making the GNU/Linux partition smallest made sense to me because it will be the only OS able to read all the other partitions so it will be able to access all the files, wherever located. So, probably all my photos and mp3s are going to be on the OS X partition.)

sudo diskutil resizeVolume disk0s2 60G Linux <name of linux volume> 20G "MS-DOS FAT32" <name of windows volume> 31G

3. Install Windows XP. You continue to follow the directions at OnMac except you plug in your external CD-Rom drive via USB and you place your old Windows 98SE disc (or comparable) in there. The Windows XP installer will find the external CD drive and your old copy of Windows without you having to do anything, except maybe press enter. Stop following OnMac’s directions just before you get to installing GNU/Linux.

When Windows boots for the first time, use your Windows XP drivers disc to add the drivers you need. Then add this Apple Mouse Utility to get right-clicking with CTRL. You’ll probably want that in your Startup folder.

You’ll also probably want a program to remap your keyboard. Currently, I don’t know how to make all the keys on the MacBook work under Windows, but I decided that a true DELETE key and a page-down key were important enough to me to sacrifice my right Apple key and that stupid little extra enter key. The linked “remapkey” program will allow that.

You now have a dual-boot system!

4. Install the rEFIt bootloader. Download the latest rEFIt (I used 0.7). Mount the disk image, copy the efi folder to your root directory, and run the enable script. That is, you’ll need to open a terminal in OS X and type:

sudo mv /Volumes/rEFIt/efi /efi
cd /efi/refit
./enable.sh

I would try booting into both operating systems a couple times. I didn’t do this step at this point, and think it would have gone more smoothly if I had. So, I’m asking you to save yourself some trouble: do it!

Update: I now believe that if you run the gptsync tool that rEFIt provides then the Debian installation (particularly the lilo part) will go much more smoothly. You need to get your EFI and MBR in sync before going to the next step. I didn’t do this at first and think that’s why lilo was such a pain to get working.

5. Install Debian Etch. The Sarge 3.1r2 Net Install disc will not work because it lacks the Marvell Gigabit Ethernet driver you need and because it won’t recognize the SATA hard drive in the MacBook. I used the Etch 2006-05-18 nightly which is already gone from the ftp site. You should get the latest Etch nightly Net Install ISO (Typically under 128Mb). Use that second CD-R to burn the iso and set up somewhere with a fast ethernet connection.

A 2.6 kernel boots by default, but I typed “expert26″ at the boot prompt anyway. I think you could just type “expert” but in any event you need to do the expert version of the install, not the normal one you get by just hitting enter.

Be sure you only format your third (empty) partition! I chose ext3, mount-point /, and flagged the partition bootable. You should select CONTINUE when it warns you that you don’t have a swap partition. We’ll make a swap file later.

Choose the 2.6.15 kernel or whatever the latest one is that your Etch installer provides.

During package selection, let tasksel install the desktop environment and the standard system (The top and bottom options, I think). You can also save yourself a lot of trouble if you add the following packages which we’ll need to compile a new kernel (among other things) later (Unfortunately none of your network connections are going to work on the first boot of Etch, so install these packages NOW):

kernel-package libncurses5-dev tk8.4 tcl8.4 module-assistant

When x.org is being configured choose 1280×800 and everything below that. We’ll also have to make some slight modifications to our /etc/X11/xorg.conf on reboot (described below).

The installer will fail when it comes time to install LILO. That’s OK. Let it try to install into your Linux partition and fail, but don’t mess with the MBR! Then go to a shell from within the installer (the option at the bottom) and type

nano /target/etc/lilo.conf

Comment out the stuff about Windows XP at the end of the file. rEFIt is handling that. Then:

chroot /target
/sbin/lilo -v

I had a good bit of trouble at this step. Keep at it until LILO is happy (although eventually if nothing works just move on and rEFIt will probably fix things later.) Your /etc/lilo.conf probably just needs to say this:

boot=/dev/sda3
root=/dev/sda3
map=/boot/map
delay=20
default=Linux
image=/vmlinuz
        label=Linux
        read-only
        initrd=/initrd.img

Hopefully because you already installed rEFIt you’ll be allowed to boot your new Debian installation right away. I wasn’t able to do this because the partition table was inconsistent. If you get strange behavior from rEFIt (I had it where selecting Tux would boot Windows, and where selecting Tux or Win wouldn’t boot anything, etc.) then run the gptsync tool that rEFIt provides and you’ll probably see that the GPT partition record doesn’t match the MBR. Let gptsync work its magic and then try rebooting all your OSs a few times, especially OS X. I think a series of reboots finally fixed things for me. I had just run the gparted liveCD just to take a look at the partitions, but I didn’t change a thing with gparted. Nonetheless, when I rebooted right after running that liveCD my Debian partition booted for the very first time via rEFIt. I’m slightly stumped as to what finally made it all work.

6. Add a 1GB swapfile. Once you finally boot into Debian, type:


dd if=/dev/zero of=/swap bs=1024 count=1048576
mkswap /swap
swapon /swap
chmod 600 /swap

Edit /etc/fstab to include:

/swap       none      swap     sw      0      0

You can type “top” to have a look at the swap and see whether it’s being used.

7. Improve the /etc/X11/xorg.conf with a few edits. This Modeline and sync rates are what an Ubuntu CD automatically generated. It may ruin our LCDs, so I warn you that these instructions are provided AS IS without warranties of any kind and with all faults.

Make changes like so:

...
Section "Device"
        Identifier      "Generic Video Card"
        Driver          "vesa"
        BusID           "PCI:0:2:0"
EndSection

Section "Monitor"
        Identifier      "Color LCD"
        Option          "DPMS"
        HorizSync       28-64
        VertRefresh     43-60
        Modeline "1280x800@60" 83.91 1280 1312 1624 1656 800 816 824 841
EndSection
...

Also make sure that the Screen section lists 1280×800 as the first option for all your depths. (I don’t know how this happened–probably my fault–but at first I had 1200×800 instead, cheating myself out of 80×800 pixels!) UPDATE: I was never really getting 1280×800 until I started using 915resolution. (apt-get install). For using an external monitor see this config file on the mactel-linux-users mailing list.

8. Compile new Linux kernel with support for Marvell Gigabit Ethernet. Hopefully you installed everything you needed to compile a kernel during installation, if not, now is when that extra CD-R or a USB key will be used. You can get needed packages for kernel compilation on another machine (or OS on the MacBook) and transfer them to your new installation. In any event you need a new Linux kernel. I used 2.6.16.18, the latest stable version as of this writing. So put at least that on your USB key or CD-R, copy it to /usr/src and then compile the kernel the Debian way:

tar -jvxf linux-2.6.16.18.tar.bz2
rm linux
ln -s linux-2.6.16.18 linux
cd linux
make menuconfig
# be sure to build in SATA and ext3 support or your kernel won’t boot. You’re also looking for the Yukon Gigabit Ethernet driver known as sky2. You should compile for Pentium-M processors and include SMP support. RTC-dev must be built-in on Core Duos in order for your hardware clock to work right. See this rtc thread. You can see my (bloated) kernel config.

make-kpkg clean
make-kpkg --append-to-version=.060525 kernel_image
# 060525 is the date in YYMMDD format

cd /usr/src
dpkg -i linux-image-2.6.16.18.060525_10.00.Custom_i386.deb
# Note again that your YYMMDD version will likely differ.

Edit your /etc/lilo.conf file to reflect the changes. Mine now looks like this:

boot=/dev/sda3
root=/dev/sda3
map=/boot/map
prompt
delay=100
timeout=100
default=Linux-2.6.16.18

image=/vmlinuz
        label=Linux-2.6.16.18
        read-only

image=/vmlinuz.old
        label=Linux-2.6.15
        read-only
        initrd=/initrd.img.old

Remove the lines “prompt” and “timeout=100″ if you just want it to automatically boot the default kernel.

RUN /sbin/lilo -v OR YOUR MacBook MAY NOT BOOT!

Cross your fingers and reboot.

9. Add Linux wireless support.
Once your ethernet interface is working again, include the following in your /etc/apt/sources.list to get the needed wireless driver. (Once wireless works, you might want to comment it out so that no future upgrade ever breaks it.)

#NEW Archive for pkg-madwifi snapshots GPG key: 71409CDF
deb http://debian.tu-bs.de/project/kanotix/unstable sid madwifi
deb-src http://debian.tu-bs.de/project/kanotix/unstable sid madwifi

apt-get update
apt-get install madwifi-source madwifi-tools

You want to install madwifi-ng the Debian way using module-assistant. Their instructions work beautifully (and have changed several times in recent weeks, so always refer to their instructions if something above seems broken.)

10. TO DO: Figure out how to use MacBook-specific keys like “fn”, try the Bluetooth, fiddle with infrared, and find a way to monitor CPU temp and fan speed. Sound works via headphones but not internal speaker. iSight now works. See instructions at the MacBook entry of the Debian wiki. Adding an external mouse provides right-click or use xmodmap to assign the eject key to right-click. Haven’t even tried the Bluetooth/infrared yet.

You can read my output from lspci, cpuinfo and cpufreq on the mactel-linux wiki.

06 MarAnother way MythTV is (and can be) better than Tivo

Sounds like somebody needs to build themselves a MythTV. Joe writes…

I just read TiVo’s Privacy Policy and it’s quite arcane. Further, trying to change your privacy preferences isn’t easy. Begin bitchy rant now…

Actually, I’ve thought that almost the opposite of the problem Joe describes (that of personally-identifiable viewing information, PIVI, being collected) faces MythTV. I think it’s extremely interesting that Tivo can tell us that ice skaters falling are the most rewound/re-watched moments of the Olympics or that people are watching The Daily Show with Jon Stewart more than anything else during the week. See Tivo Central. That’s valuable sociological/pop-culture data that isn’t captured for MythTV users.

It seems easy enough to solve since MythTV is a free software project. We can easily get the interesting data and avoid the privacy concerns: MythTV developers (or you!) could write a plugin that allows MythTV users to opt-in to anonymized data collection and then a network of volunteer servers could collect and display the results. The plugin would also be free software and those with privacy concerns could inspect the code and be assured that any personally-identifiable data was being stripped out. However, since MythTV is both free and no-cost, there isn’t billing data such as address and credit card information to begin with. The only personally-identifiable info that could possibly be collected would be IP address and it’d be easy to write the plugin so that it required Tor to anonymize even that.

This viewing data is interesting in so many ways. I think what shows a person records may say a lot about them. Knnowing what shows most people are recording also lets you know if you’re missing something worth checking out. I’ve even thought that a match-making/dating service would benefit from collecting individualized recording data and then matching people based on it. Even if romance didn’t develop, they could certainly agree on what to watch. (I’m publishing this last idea now in the hopes of thwarting any future business-method patent based on it.)

29 JanCompare Two Directories from the Command Line

UNIX apparently has a dircmp program that will compare two directories and tell you where they differ. This is useful if you want to ensure that two directories are identical or nearly so. Maybe I’m looking in the wrong places, but I can’t find dircmp for Debian. Instead, I think this works:

diff -r --brief dir1 dir2

Enjoy.

19 JanFirefox Add-on Engines A-plenty

1. You should be using Firefox to browse the web.

2. One reason is how many add-on search engines you can add to the top right corner so that searching Wikipedia, Merriam-Webster, eBay, Amazon, Flickr Tags, etc is just a click away and built right into the browser. It’s stunningly useful. Begin with Mozilla’s 23 popular add-on search engines, but then check out the motherload of add-on search engines at mycroft. Just click on ‘Google’ alone and see all the different ways you can more easily utilize Google (searching News, Images, etc.) and it’ll blow your mind.

3. Note: I’ve found that after installing a bunch of these at once sometimes Firefox needs a restart before it sorts all your changes. So, if you notice strange behavior, don’t panic, just restart.

09 JanThe Even More Perfect Debian Sarge Setup

Falko Timme at howtoforge.com has a number of excellent howtos on setting up GNU/Linux servers. I have learned much from using his Perfect Setup for Debian Sarge 3.1.

However, when I finish his perfect setup, there remain a few things that I think essential to do, particularly regarding security. There are an increasing number of dictionary attacks against ssh servers that should be addressed. I do the following:

# apt-get install logcheck
edit /etc/logcheck/logcheck.conf to change the SENDMAILTO=”your@email.address” line to include your email address so you can be notified of suspicious log activity.

To actually ban those ssh attackers, I love a program called fail2ban that is currently in Debian unstable, but not in stable. I prefer to install it without messing with my apt sources by browsing ftp://ftp.debian.org/debian/pool/main/f/fail2ban/ and noting the filename of the latest version. Then

# wget ftp://ftp.debian.org/debian/pool/main/f/fail2ban/fail2ban_0.6.0-2_all.deb
# dpkg -i fail2ban_0.6.0-2_all.deb

Then I edit /etc/fail2ban.conf and change the maximum failures allowed from 5 to 3 and the time (in seconds) that the failed IP is banned from 10 minutes to a little over two months. Also, set up the section entitled [MAIL] to notify you of the bans.

maxfailures = 3
bantime = 6000000

[MAIL]
enabled = true
from = fail2ban@your.domain.com
to = your@email.address

# /etc/init.d/fail2ban restart

Next, edit /etc/ssh/sshd_config and add the usernames of anyone authorized to have ssh access:

AllowUsers user1 user2 user3

Then, perhaps it’s the nascent attorney in me, but I like to put the SSH attackers on notice that their unauthorized access attempts are not welcome. In the same
sshd_config file, uncomment

Banner = /etc/issue.net

Then edit /etc/issue.net to contain only the following text:

If you are not authorized to access this system, LEAVE NOW.
Access attempts will be logged. Unauthorized access will be prosecuted.

On servers that have excess processor cycles and bandwidth, it’s also nice to help out the Tor network by (at least) being a middle-man server. (Tor is free software that improves your privacy online and their network relies on volunteer servers.) I prefer to run the latest version and stay up to date, so modifying /etc/apt/sources.list is the way to go. Add:

deb http://mirror.noreply.org/pub/tor experimental-0.1.1.x-sarge main

# apt-get update
# apt-get install tor privoxy socat

Then to allow no more than 1 GB of traffic per day at an average rate no greater than 75 KB/s edit /etc/tor/torrc like so:

Nickname something-unique-like-your-hostname
ContactInfo Your Name <your AT email dot address>
BandwidthRate 75 KB
AccountingStart day 12:00
AccountingMax 1 GB

Then be sure to uncomment:

ORPort 9001
DirPort 9030
ExitPolicy reject *:* # middleman only — no exits allowed

# /etc/init.d/tor restart

Then make logcheck do a little more work for you by editing /etc/logcheck/logcheck.logfiles to include:

/var/log/tor/log
/var/log/daemon.log

After a few days, when you know the tor server is working correctly, you should register it. Send mail to tor-ops@freehaven.net with a subject of ‘[New Server] (your server’s nickname)’ and include the following information in the message:

  • Your server’s nickname
  • The fingerprint for your server’s key (the contents of the “fingerprint” file in your DataDirectory — on Linux/BSD/Unix, look in /var/lib/tor or ~/.tor)
  • Who you are, so the tor ops know whom to contact if a problem arises
  • What kind of connectivity the new server will have

Finally, you should implement some sort of backup process. Falko at howtoforge comes to the rescue again with his Automated backups with rdiff-backup. (Strangely, I can get remote backups to work like this fine, but backing up the backup server itself required me to resort to a root cronjob, despite different howtos describing two alternative ways to handle this. –Update: solved.)

Also, if you’re doing rdiff-backups across various GNU/Linux distributions it’s usually important to have the same version of rdiff-backup installed on each. In this event, you might not want to follow the installation instructions at howtoforge (just the subsequent configuration stuff). For manual installations, do this:

Step 1: Get Python dependencies (explained for Debian, but just do the equivalent for your distro).

# apt-get install python2.3 python2.3-dev python2.3-pylibacl python2.3-pyxattr

(Those last two are optional, but you might as well…)

Step 2: Get librsync.

# wget http://easynews.dl.sourceforge.net/sourceforge/librsync/librsync-0.9.7.tar.gz
# tar zvxf librsync-0.9.7.tar.gz
# cd librsync-0.9.7
# ./configure
# make
# make install
# ldconfig

Step 3: Get rdiff-backup.

# wget http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-1.0.4.tar.gz
# tar zvxf rdiff-backup-1.0.4.tar.gz
# cd rdiff-backup-1.0.4
# python setup.py install

Then you configure according to the howtoforge article linked above and you’ll be backing up in style.

When I get a chance I may also explain how to set up snort, portsentry, and spamassassin. I’ve also used Bastille in the past. For the security-conscious, that’s worth looking into as well.