A Tool Package for the Promise NS4300N

Picking the tools for the package

Building the ncurses 5.7 Library

This was a little trickier than I expected because the library has some internal dependencies that make cross-compilation very difficult. The library depends on a compiled terminfo directory from the included terminfo.src. The terminfo directory created by a binary utility in the package called tic. However, the previous versions of tic have a bug exposed by the 5.7 version of the terminfo.src. The package detects a cross-compile and attempts to use the host's version of tic, which will hang forever if it is not version 5.7. To solve the chicken and egg problem here, my build script for ncurses-5.7 builds the code three seperate times. The first time builds a version that will execute on the host system, placing it into a temporary directory. Then the cross-compile environment is used to build it twice more: once to update the cross-compile environment itself and once for the target environment. Why build it twice? Because ncurses hard codes absolute paths into the library.

If any one can think of a better way to do this, I am open to suggestions. Another choice I made was to truncate the terminfo database from over 1500 elements to the most useful 33. My thinking is that there are very few h19 or adm3a terminals left in operation, and they are unlikely to be logging into the NAS. Here is my build script ncurses_doit.sh for the library:

#!/bin/sh # The ncurses library also build serveral binaries, one of which # (tic) is then used to compile the terminfo.src file into the # termcap library. Someone found a bug in tic via the terminfo.src # file, causing tic to loop forever. The problem is you need tic # v5.7 to install terminfo v5.7, so it's a chicken and egg problem. # So here I'll build a copy of the binaries on the host machine, # then use them to build for the target. # These are important for tools built by the Makefile for execution on host BUILD_CC=`which gcc` BUILD_AS=`which as` gcc_dir=${BUILD_CC/%\/gcc/} as_dir=${BUILD_AS/%\/as/} BUILD_CFLAGS="-B$gcc_dir -B$as_dir" BUILD_LDFLAGS="-L/lib -L/usr/lib" TEMP_DIR=`pwd` TEMP_DIR=${TEMP_DIR}/tmp # First, I'm going to reduce the number of entries in the termlib # to only the 33 most common and useful entries patch -N misc/run_tic.in <<"EOF" --- run_tic.in.org 2006-10-28 09:43:30 +++ run_tic.in 2009-04-30 23:10:17 @@ -112,3 +112,3 @@ EOF -if ( $SHLIB tic$suffix -x -s -o $TERMINFO $source ) +if ( $SHLIB tic$suffix -x -s -e hurd,ansi,pcansi,dumb,wsvt25,wsvt25m,mach-bold,mach-color,mach,vt102,vt220,vt52,vt100,Eterm,linux,xterm-mono,xterm,xterm-r6,xterm-r5,xterm-vt220,xterm-debian,xterm-xfree86,xterm-color,rxvt-unicode,rxvt-basic,rxvt,screen-s,screen-bce,sun,screen-w,screen,cons25,cygwin -o $TERMINFO $source ) then @@ -131,3 +131,3 @@ EOF -if ( $SHLIB tic$suffix -s -o $TERMINFO $source ) +if ( $SHLIB tic$suffix -s -e hurd,ansi,pcansi,dumb,wsvt25,wsvt25m,mach-bold,mach-color,mach,vt102,vt220,vt52,vt100,Eterm,linux,xterm-mono,xterm,xterm-r6,xterm-r5,xterm-vt220,xterm-debian,xterm-xfree86,xterm-color,rxvt-unicode,rxvt-basic,rxvt,screen-s,screen-bce,sun,screen-w,screen,cons25,cygwin -o $TERMINFO $source ) then EOF ./configure --prefix=$TEMP_DIR # Make the binary of "file" that will execute on the build machine make install.includes install.progs # Now clean out all traces of the build to avoid contamination make distclean BASE=~/src/crosstool/gcc-3.4.3-glibc-2.3.2/powerpc-linux-gnu PACKAGE=`pwd`/../plugin/tools/TOOLS PACKAGE_DATA=/data/usr/share PACKAGE_LINK=/data/usr/lib export CC=powerpc-linux-gnu-gcc export AR=powerpc-linux-gnu-ar export PATH=$TEMP_DIR/bin:$BASE/bin:$BASE/powerpc-linux-gnu/bin:$PATH # Configure, make and install into our build environment ./configure --prefix=$BASE/powerpc-linux-gnu \ --libdir=$BASE/powerpc-linux-gnu/lib \ --includedir=$BASE/powerpc-linux-gnu/include \ --with-build-cc=$BUILD_CC \ --with-build-cflags="$BUILD_CFLAGS" \ --with-build-ldflags="$BUILD_LDFLAGS" \ --host=powerpc-linux-gnu \ --with-shared --with-normal --without-debug make make install.libs install.includes install.ncurses install.progs install.data make clean # Configure, make and install into our target environment ./configure --with-terminfo-dirs=$PACKAGE_DATA \ --datadir=$PACKAGE_DATA --prefix=/ \ --libdir=/lib \ --includedir=/include \ --with-build-cc=$BUILD_CC \ --with-build-cflags="$BUILD_CFLAGS" \ --with-build-ldflags="$BUILD_LDFLAGS" \ --host=powerpc-linux-gnu \ --with-shared --with-normal --without-debug make make DESTDIR=$PACKAGE install.libs install.includes install.ncurses install.progs mkdir -p $PACKAGE/$PACKAGE_LINK make DESTDIR=$PACKAGE install.data

Building the file package

The file utility is able to examine a file to determine the type of data is in the file. It's not perfect but it's better than "the brand X" method of examining the file name extension and declaring "Ta da!" However, file relies on a magic number database that attempts to decode the first few bytes of a file's contents. As shipped in the package, the database is a text file, but as part of the build process, the generated file binary is used to compile the database. Again, as with ncurses, the file binary and text magic number database format are tightly coupled. So my build script file_doit.sh again builds a local binary, then a target binary package (editorial comments have been deleted):

#!/bin/sh TEMP_DIR=`pwd` TEMP_DIR=${TEMP_DIR}/tmp export PATH=$TEMP_DIR/bin:$PATH ./configure --prefix=$TEMP_DIR # Make the binary of "file" that will execute on the build machine make install # Now clean out all traces of the build to avoid contamination make distclean BASE=~/src/crosstool/gcc-3.4.3-glibc-2.3.2/powerpc-linux-gnu export CC=powerpc-linux-gnu-gcc export AR=powerpc-linux-gnu-ar export PATH=$BASE/bin:$BASE/powerpc-linux-gnu/bin:$PATH # This song and dance necessary because included libtools bites PACKAGE=`cd .. ; pwd` PACKAGE=${PACKAGE}/plugin/tools/TOOLS TARGET_DATA=/data/usr/share ./configure --prefix=/ --libdir=/lib \ --includedir=/include \ --datarootdir=$TARGET_DATA \ --mandir=$TARGET_DATA/man \ --host=powerpc-linux-gnu make make DESTDIR=$PACKAGE install

Building the other packages

The rest of the packages are relatively straight forward to build and can be found here: bash_doit.sh, gawk_doit.sh, grep_doit.sh, less_doit.sh, pcre_doit.sh, sed_doit.sh, tcsh_doit.sh, wget_doit.sh, and which_doit.sh. Boy, that seems like alot of work. Can't we make this process any easier? Yes we can.

Automating the build

I built a script that I not terribly imaginatively called build_script.sh. This tool will automate not just the building of the packages, but also the downloading of the the source code packages, and the creation of the NS4300N plugin package.

#!/bin/sh # $PLUGIN_NAME must match PKGNAME in the file "rev" PLUGIN_NAME=tools PLUGIN_VERSION=010000 PLUGIN_FILE=${PLUGIN_NAME}_${PLUGIN_VERSION}.ppg PLUGIN_DIR=plugin PKG_DIR=${PLUGIN_NAME} TOP_DIR=${PLUGIN_DIR}/${PKG_DIR} TREE_DIR=$TOP_DIR/TOOLS SBIN_DIR=$TREE_DIR/sbin # These are (at the time of writing) the locations of the most current # source packages of these utilities. To drop any of them (I suggest # that you at least build the first two which are libraries) just # delete or comment out its definition line. To add a new package, # you can add a new XYZ_SRC variable, add it to the build list, and make # an XYZ_doit.sh script to this directory. NCURSES_SRC="http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.7.tar.gz" PCRE_SRC="ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.9.tar.gz" BASH_SRC="http://ftp.gnu.org/gnu/bash/bash-4.0.tar.gz" FILE_SRC="ftp://ftp.astron.com/pub/file/file-5.00.tar.gz" GAWK_SRC="ftp://mirrors.kernel.org/gnu/gawk/gawk-3.1.6.tar.gz" GREP_SRC="ftp://mirrors.kernel.org/gnu/grep/grep-2.5.4.tar.gz" LESS_SRC="http://www.greenwoodsoftware.com/less/less-429.tar.gz" SED_SRC="ftp://mirrors.kernel.org/gnu/sed/sed-4.1.5.tar.gz" TCSH_SRC="ftp://ftp.astron.com/pub/tcsh/tcsh-6.16.00.tar.gz" WGET_SRC="http://ftp.gnu.org/gnu/wget/wget-1.11.4.tar.gz" WHICH_SRC="ftp://mirrors.kernel.org/gnu/which/which-2.20.tar.gz" BUILD_LIST="${NCURSES_SRC-} ${PCRE_SRC-} ${BASH_SRC-} \ ${FILE_SRC-} ${GAWK_SRC-} ${GREP_SRC-} \ ${LESS_SRC-} ${SED_SRC-} ${TCSH_SRC-} \ ${WGET_SRC-} ${WHICH_SRC-}" if test "${1+set}" = set; then if test $1 = clean; then \rm .*.done > /dev/null 2>&1 \rm -rf $PLUGIN_DIR > /dev/null 2>&1 \rm $PLUGIN_FILE > /dev/null 2>&1 exit 0 fi fi # Build the destination directory mkdir -p $SBIN_DIR for PROJECT_SOURCE in $BUILD_LIST ; do # Extract the archive file name by stripping the URL off PROJECT_FILE=${PROJECT_SOURCE/#*\/} # Exract the directory name by stripping off the archive extension PROJECT_DIR=${PROJECT_FILE/%.tar.gz/} # Extract the project name by stripping off version info PROJECT_NAME=${PROJECT_DIR/-[.0-9]*/} # Generate the name of the config & make script for project PROJECT_DOIT=${PROJECT_NAME}_doit.sh # Generate at sentinel name to check if project completed SENTINEL=.${PROJECT_DIR}.done if test ! -f ${SENTINEL}; then echo echo "************************************************************************" echo "************************* Building $PROJECT_DIR ************************" echo "************************************************************************" (test -f $PROJECT_FILE ) || wget $PROJECT_SOURCE || {(echo "COULD NOT FIND SOURCE FOR $PROJECT_DIR" ; exit 1;); exit 1;} (test -d $PROJECT_DIR ) || tar -xvzf $PROJECT_FILE || {(echo "COULD NOT EXTRACT $PROJECT_DIR" ; exit 1;); exit 1;} cp $PROJECT_DOIT $PROJECT_DIR/doit.sh if test ${PROJECT_NAME} = tcsh; then cp tcsh_ns4300.defs $PROJECT_DIR/ns4300.defs fi cd $PROJECT_DIR ./doit.sh || {(echo "BUILD FAILED" ; exit 1;); exit 1;} cd .. \rm -rf $PROJECT_DIR # \rm $PROJECT_FILE touch ${SENTINEL} echo "####################### Built $PROJECT_DIR Successfully ##################" fi done echo echo "####################### Built All Projects Successfully ##################" cp rev $PLUGIN_DIR cp upgrade_script $TOP_DIR chmod 755 $TOP_DIR/upgrade_script cp plugin.conf $TREE_DIR chmod 755 $TREE_DIR/plugin.conf cp tools_start $SBIN_DIR chmod 755 $SBIN_DIR/tools_start sudo chown -R root:root $TOP_DIR $PLUGIN_DIR/rev tar -czf ${PLUGIN_NAME}.tar.gz -C $PLUGIN_DIR $PKG_DIR rev sudo chown -R $USER:$GROUP $TOP_DIR $PLUGIN_DIR/rev dd if=/dev/zero of=$PLUGIN_FILE bs=1k count=97 cat ${PLUGIN_NAME}.tar.gz >> $PLUGIN_FILE \rm ${PLUGIN_NAME}.tar.gz

The plugin package and installation

The build script creates a plugin with a directory structure that looks like this:

plugin │ ├──rev (revision information) │ └──tools │ ├──upgrade_script (one time use, executable) │ └──TOOLS │ ├──plugin.conf (configuration information) │ ├──data │ │ │ └──lib (ln -s /data/usr/lib) │ │ │ └──share (ln -s /data/usr/share) │ ├──sbin │ │ │ └──tools_start (service start script, executable) │ ├──bin (ln -s * /bin) │ ├──etc (ln -s * /etc) │ ├──lib (ln -s * /lib) │ └──libexec (ignored)

The installation process took some time to arrive at. The /usr file system is a squashfs in FLASH memory, whish is read only. And the root file system is a 15MB ext2 file system on volatile RAM memory, with only about 5MB available. So the obvious choice is to leave the plugin files in the /VOLUMEn/PLUGINAPP/TOOLS directory. Then one approach is to modify the PATH and LD_LIBRARY environment variables to point to them. I chose to instead place symbolic links in the root file system to point into the plugin directory. However, this can lead to problems of clobbering existing utilities that can not be undone if you choose to remove the plugin, without rebooting the NAS. I chose then to copy items that would be clobbered into a backup directory hidden in the target directories. This is all handled by the tools_start script, executed when the package is initialized:

#!/usr/bin/perl # Promise has installed Perl 5.004 which is 12 years old, and I have # version 5.10 on my development machine. Imagine my suprise. $old_perl = 1; $chmod_cmd = "/bin/chmod"; $df_cmd = "/bin/df"; $ln_cmd = "/bin/ln"; $mv_cmd = "/bin/mv"; $mkdir_cmd = "/bin/mkdir"; $rm_cmd = "/bin/rm"; $rmdir_cmd = "/bin/rmdir"; $root_dir = "/"; $profile = $root_dir . "etc/profile"; $profile_tmp = $profile . "tmp"; $terminfo = "/data/usr/share/terminfo"; $terminfo_env = "export TERMINFO=$terminfo\n"; $bak_dir_name = ".tools.BAK"; @dir_list = ( "bin", "lib", "etc", "data/usr" ); $action = $ARGV[0]; $app_path = $ENV{'APP_PATH'}; # search installed path if ( $app_path eq "" ) { open(IN, "$df_cmd |"); while(<IN>){ if (/(VOLUME\d+)/) { if ( -d "/$1/PLUGINAPP/TOOLS" ) { $app_path = "/$1/PLUGINAPP/TOOLS"; last; } } } close(IN); } if ( $app_path eq "" ) { die "Can't find the installation path!"; } # This code "installs" the contents of directories in the array @dir_list # from the $app_path to the $root_dir, by directory. No files are coppied # from $app_path, only symlinks are created within the $root_dir directories # to the $app_path directories. When there is a name collision, i.e., there # is an element $app_path/ABC/XYZ and $root_dir/ABC/XYZ, then a directory # named $bak_dir_name is created in $root_dir/ABC, and the original file is # safely stored there: eg. as $root_dir/ABC/$bak_dir_name/XYZ. Then the # symlink is created as normal. # # When run with the parameter "stop," the process is reversed, the symlinks # are removed from $root_dir, any files stored in any $bak_dir_name are # restored and the $bak_dir_name directories are removed. The removal of # symlinks checks to make sure that they still point to the $app_dir; if # they do not or they are not a symlink, then they are left as is. if ( $action eq "start") { # print "Starting up\n"; foreach $dir_name ( @dir_list ) { # make symbolic links from /$dir_name to our $app_path/$dir_name $app_dir = "$app_path/" . $dir_name; $dest_dir = $root_dir . $dir_name; opendir( DIR_HANDLE, $app_dir ); @file_list = readdir( DIR_HANDLE ); closedir( DIR_HANDLE ); foreach $file_name ( @file_list ) { # print " Working on file $file_name\n"; $full_pathname = "$app_dir" . "/" . "$file_name"; $link_pathname = "$dest_dir" . "/" . "$file_name"; next if (($file_name eq ".") || ($file_name eq "..")) ; if (( -f $link_pathname ) || ( -l $link_pathname )) { # print " Found existing $link_pathname - "; # Check for existing link - eg. we were started 2X if ( -l $link_pathname ) { $temp = readlink $link_pathname; if (( readlink $link_pathname ) =~ m/$app_dir/ ) { # print " Hey, it's our link!\n"; next; } } # Else, it is not ours, save it into a back up dir $back_dir = "$dest_dir" . "/" . $bak_dir_name; if (! -d $back_dir ) { # Create it if need be make_dir( $back_dir, 0755 ); } $back_pathname = $back_dir . "/" . $file_name; # print " Moving it to $back_pathname\n"; move_file( $link_pathname, $back_pathname); } # print " Linking $link_pathname to $full_pathname\n"; link_file( $full_pathname, $link_pathname ); } } # Now we fix the TERMINFO environment variable. It should have been # compiled into ncurses when I built it, but that doesn't seem to be # working. $missing = 1; open PROFILE, "+<$profile"; while(<PROFILE>) { if ( $_ eq $terminfo_env ) { $missing = 0; last; } } if ( $missing ) { print PROFILE $terminfo_env; } close PROFILE; exit 0; } elsif ($action eq "stop") { # print "Stopping\n"; # Remove symbolic links from our /$dir_name to $app_path/$dir_name foreach $dir_name ( @dir_list ) { $app_dir = "$app_path/" . $dir_name; $dest_dir = $root_dir . $dir_name; opendir( DIR_HANDLE, $app_dir ); @file_list = readdir( DIR_HANDLE ); closedir( DIR_HANDLE ); foreach $file_name ( @file_list ) { $link_pathname = "$dest_dir" . "/" . "$file_name"; if ( -l $link_pathname ) { $full_pathname = readlink $link_pathname; if ( $full_pathname =~ m/$app_dir/ ) { # print " Unlinking $link_pathname to $full_pathname\n"; del_file( $link_pathname ); } else { # print " Not unlinking $link_pathname\n"; } } } $back_dir = "$dest_dir" . "/" . $bak_dir_name; if ( -d $back_dir ) { opendir( DIR_HANDLE, $back_dir ); @file_list = readdir( DIR_HANDLE ); closedir( DIR_HANDLE ); foreach $file_name ( @file_list ) { next if (($file_name eq ".") || ($file_name eq "..")) ; $back_pathname = "$back_dir" . "/" . "$file_name"; $link_pathname = "$dest_dir" . "/" . "$file_name"; # print " Restoring $back_pathname to $link_pathname\n"; move_file( $back_pathname, $link_pathname ); } # It had better be empty at this point del_dir( $back_dir ); } } # Now we unfix the TERMINFO environment variable. $missing = 1; open IN, "<$profile"; open OUT, ">$profile_tmp"; while(<IN>) { next if ( $_ eq $terminfo_env ); print OUT; } del_file( $profile ); move_file( $profile_tmp, $profile ); exit 0; } sub move_file { my $from_file = $_[0]; my $to_file = $_[1]; if ( $old_perl ) { system("$mv_cmd $from_file $to_file > /dev/null 2> /dev/null "); } else { ( link $from_file, $to_file ) || die "Move couldn't hard link $from_file to $to_file\n"; ( unlink $from_file ) || die "Move couldn't unlink $from_file\n"; } } sub make_dir { my $dir_name = $_[0]; my $dir_mode = $_[1]; if ( $old_perl ) { system("$mkdir_cmd $dir_name > /dev/null 2> /dev/null "); system("$chmod_cmd $dir_mode $dir_name > /dev/null 2> /dev/null ") } else { mkdir $back_dir, $dir_mode; } } sub link_file { my $file_name = $_[0]; my $link_name = $_[1]; if ( $old_perl ) { system("$ln_cmd -s $file_name $link_name > /dev/null 2> /dev/null "); } else { (symlink $file_name, $link_name) || die "Couldn't symlink\n"; } } sub del_file { my $file_name = $_[0]; if ( $old_perl ) { system("$rm_cmd $file_name > /dev/null 2> /dev/null "); } else { unlink( $file_name ); } } sub del_dir { my $dir_name = $_[0]; if ( $old_perl ) { system("$rmdir_cmd $dir_name > /dev/null 2> /dev/null "); } else { rmdir( $dir_name ); } }