Input redirection into a while loop

I'm teaching shell scripting in Stamford this week. Earlier today we had a discussion about data redirection to/from various programming statements like if, while, etc.

The offered approach was:

while read -r
do
command(s)
done 0< file

One of the students asked if you could have the redirection specified before the while, i.e:

0< file while read -r
do
command(s)
done

My immediate thought was - yes, it should work. I didn't offer my opinion at that point, though. Instead I answered the question with another one. I asked them if the following would work (my favourite example illustrating shell scan):

$ 1> file < /etc/passwd cat 2> /dev/null

I was truly convinced that the original question is another exhibition of the same behaviour, which allows to specify the I/O redirection at any (sensible) point of the command line.

Well, whereas the cat command line above will work OK, the original construct does not work! I have tried various combinations, and deduced that any alias, function or external command can have redirection specified before the command itself. However, when you are dealing with reserved words (programming keywords) or built-ins, the mechanism is not recognised, with a "command not found" error message.

Curious... Need to investigate it further, unless Clive comes back with an answer before then ;-)

URL code/decode tool

NLUG discussion on debugging htmp/php code lead to one of the participants quoting what I thought was an excellent tool for url decoding.

There are also very useful conversion tables.

Mac addresses in VMWare

I have recently noticed that when booting RHEL 4 in VMWare 5 (on my HPnc6120 laptop) the eth0 card would not initialise, with an error "MAC address is different than expected - failed".

After some googling, one theme appeared to recur: hardware modules in the kernel problem. However, I was convinced this was not the issue in my case, since I also have FC 5 and SuSE, to compare the kernel config files against. To complicate the matter further, I went down the totally wrong way to start with, assuming that the correct value was one reported by ip neigh show from another machine on the same home network. How silly of me - that other host was looking at the 'front end' configuration, whereas my problem was sitting behind the VM...

Eventually I managed to fix the problem, but without really understanding the reasons. What was the fix?

Well, I noticed that the HWADDR parameter in the /etc/sysconfig/network-scripts/ifcfg-eth0 had a different value than the one reported by the ip address show command.

I changed the HWADDR value to the one reported by the ip command, restarted the network with service network restart, and all was well. But I couldn't understand why (and how) the configuration file got changed - I could swear that I hadn't touched the parameter - ever! So I continued googling - and eventually went directly to the vmware site (another lesson learnt - should have gone there much sooner!) . The answer is well documented amongst their support pages, under "Maintaining and Changing the MAC Address of a Virtual Machine".

In a nutshell: virtual machine also means virtual MAC. The value assigned is unique, but is liable to be changed if you move the location of your VM.

This is exactly what happened to me. Since creating the RHEL4 VM, I have moved the "My Virtual Machines" folder (yes, XP) to a different drive - to fix XP synchronisation issues... But in turn , that messed up my RHEL installation.

Since my fix did the trick, I haven't studied the solutions they quote on their site - apparently you can tweak configuration files of VMWare itself to fix this.

I'm just happy that there was a 'good' reason for my problems. I hate fixing the unknown... It's just like replacing a blown fuse without knowing why it blew in the first place ;-)

Just a link... to a Microsoft blogger!

I've come across a very interesting discussion, started by a blog entry. It comes from an MS enclosure, and is a rather adventureous and brave expression of individual's views.

In the world where corporates often attempt to control the expression of views of their employees - usually through the fear of the unknown - this blog entry is proving that given a chance, most employees are prepared to stand by their bread provider, for better and for worse, in sickness and in health.

So, whether you agree with the preaching, analisys and recommendations or not - it does make interesting reading.

Linux in the air

Embedded Linux from LynuxWorks has been cleared for use in display, flight management and flight control systems in American commercial fleets.

"LynuxWorks becomes the first and only embedded operating system vendor to receive an Advisory Circular AC 20-148 acceptance letter from the FAA. "

See more at http://www.lynuxworks.com/corporate/press/2006/rsc-acceptance.php.

F-Spot - open source answer to Picassa

There is only a handful of applications which ensured that I continue using Windows. Picassa from Google (picture and video editing and management program) has definitely been one of them. Personally, I could never understand why they resist releasing a version of Picassa for Linux, but resist they do.

However, Linux community responded, as usual, and equivalent tool emerged. It's name is F-Spot, and comes from: http://f-spot.org/ F-Spot can be used on a non-Linux platform as well - if you download a Mono-Live CD , which comes with all sorts of goodies, F-Spot included.

The current version, 0.1.10, is actually quite good and stable. I haven't tested it fully yet, but it certainlyy is usable. The only feature I'm missing is video clips management, but this is very early days for the project, and developer team is keen to hear of any "shopping lists". Versioning is good, editing less comprehensive than that of Picassa's, but on the whole the time of good by to Windows is nearer and nearer.

Any QA person reading this - relax, I will maintain a system for course design applications, at least until we get to use some portable standard ;-)

New phase in X server evolution

For a while Novell seemed to head for the open source community doghouse, for working on the new X standard (Xgl) behind closed doors. They have also been developing a new window and composition manager called Combiz. Well, both of these products have now been released into Xorg repositories. So all is well. Is it? There are two issues that this has raised. One is the new wave of standard in-fighting that we are seemingly face. This is because as soon as the community accepted Novell's Xgl (or Xegl - OpenGL version) as the future for X11, Red Hat put the spanner in the works by promoting an alternative: aiglx (accelerated indirect GL X. Red Hat's argument would be sound - both Xgl and Xegl are brand new solutions, which would require a lot of work on the part of driver creators, whereas aiglx is just an extension to the existing X server standard. Hmmm. Here we go again... The other issue that occurred to me is more of the philosophical nature. It refers to the reasoning Novell offered in defense of the 'keep it behind the closed door' approach to the design. The "design by committee" concept is what most companies face, and have to deal with. Novell chose to go against the politically correct community involvement in the early stages of planning, spec'ing and designing. They are being slagged for it, but should they really. For one, I fully agree with Dan Winship's reasoning...

See: http://mail.gnome.org/archives/desktop-devel-list/2006-February/msg00115.html .

Boot from CD - a winner again

Didn't expect to write anything whilst on holidays in Sydney. But couldn't resist sharing my joy of Knoppix saving me from a big row with the hotel staff.

Hotel I'm in is using a third party company to provide telecomms and broadband services. It's awfully expensive, but convenient, so I have subscribed.

So, to get online, you tune into channel 8 on your telly, choose the deal (9.95 for an hour, 44.95 for 3 days), then enter your room number - and bingo, a password is created for you. You connect your laptop and off you go. Except that the socket in our room was faulty. A very friendly manager allowed us to use another room (to test my laptop), using a password he provided for me. It worked (I never doubted my laptop). So, we move rooms altogether. Once moved, tried to connect and the fun started.

Tried the password I had generated before - error message saying - cannot activate: already active. Although the manager's password has expired, I could still connect to the Internet, and use exactly the same sites I did as part of the test, but no others. Tried clearing the browser's cache, reboot, clear the cache again for good measure, same behaviour over and over.

MAC address tied it in? Don't know, but booted from the Knoppix CD, connected the cable, punched in the password - straight in! On the same laptop, with the same hardware.

So, if not a browser's cache, not an interface number, what was it that tied my laptop to the servers information? Am I missing something? Or is the comms provider being naughty and captures more information that I would comfortably want to surrender? I left the question with the manager. Somehow, I don't expect to get any answers before we leave in three days time...

Moral of the story - Knoppix came up trupms again!

Allowing remote X sessions

A couple of posts before, I have described how to use gdmsetup to enable (in conjunction with xhost + command) remote X applications on your machine.

I've been working with our engineers (OK, here goes the mention - Dave Williams, who is fast becoming a Linux expert, and Martin Pain - who, similarly is becoming Oracle and SQL ex if not expert), to update our various course setup documents. The fix for gdmsetup needed to be scripted. A simple trick allowed me to identify the configuration line that needed to be changed:

First save the current setting in a different file:

# cp /etc/X11/gdm/gdm.conf /tmp/x.$$

Then change the setting; run:

# gdmsetup

Select Security tab, and modify the setting for the "Always disallow TCP connections...".

Once done, just compare the two files:

# diff /etc/X11/gdm/gdm.conf /tmp/x.$$
191c191
< disallowtcp="true
---
> DisallowTCP=false

Bobs your uncle. A single line (or two):

sed '/DisallowTCP/s/false/true/' /etc/X11/gdm/gdm.conf > /tmp/x$$
mv /tmp/x$$ /etc/X11/gdm/gdm.conf

placed in an appropriate script, and X applications will now be accepted.

RPM package verification

This post doesn't solve any problems, but is more of a record of the investigation to solve one.

I'm having problems with Fedora Core 4 GUI. Rather unpredictible X behaviour, with not being able to restart it, inability to start just some of the applications, such as xterm or firefox...

Handling X has always been my achilles heal. I resisted windows, of any description ;-), for as long as I could, initially regarding it first as pure waste, then unnecessary nicety. It's only relatively recently that I kind of resigned to the fact that resources are now robust enough to cope, and there exist applications that could not live without GUI.

Anyway, still don't really know what the problem is, but done some googling, and can see that I'm not the only one struggling with this strange GUI behaviour. Somebody suggested verifying X server packages - good idea. Run the rpm --verify on all packages to do with xorg, and got some result:

# rpm -V $(rpm -qa | grep xorg)
. . ? . . . . . c /etc/security/console.apps/xserver
S . 5 . . . . T /usr/X11R6/lib/modules/libvgahw.a

So why post this? Well, mostly to document the result, and to make a record of the file attribute characters.

The eight characters mean differences in:
S - size
M - file type or permissions (mode)
5 - MD5 checksum
D - major/minor device number
L - access path (read-link)
U - user (owner)
G - group
T - mtime stamp (modification time)

Between attribute characters and the file name, the file's type (as in "purpose") may be listed. In my output I got 'c' - a configuration file.


All I know now, that the checksum on the xserver file cannot be verified and that the module archive file has a different size, mtime and checksum (considering what type of file it is - hardly surprising).

So, have I learned anything - not really, but it gave me a chance to experiment with verifying packages ;-)

Working with UNIX 'epoch'

An administrator friend was trying to find out on which date the last change of a password occured. He knew that this information was stored in the /etc/shadow file.

Well, the third field of the shadow file contains a number, which is equivalent to number of days from the epoch, which for UNIX was agreed to be January 1, 1970, at midnight, UTC.

So, let's see how we can convert this information into a meaningful date. For example, on my test machine I have:

$ grep root /etc/shadow
root:$1$.LbW0Bv2$keqd6WlAumjwvqRl2tu6U1:13117:0:120:7:::

The conversion is simpler that you may think - no complicated calculations, just yet another useful option of the date command:

$ date -d "1970-01-01 utc + 13117 days"
Wed Nov 30 00:00:00 GMT 2005

The above command means: show the date based on the given "string". The string used here says: use the epoch as the base date, and add to it the value of the 3rd field (for the user of interest) from the /etc/shadow file .

The unit at the end of the string is important! For example, consider the following:

$ date +%s
1137404710
$ date -d "1970/01/01 utc + 1137404710 sec"

Mon Jan 16 09:46:11 GMT 2006

In the above example we first convert the current date into the 'epoch' value, but expressed in seconds (notice that the +%s formatting is only available on the GNU-enhanced versions of date). We then used the date command to convert this number back into a proper date format.

You could use the output of the date command as part of the string used in the calculations. The following is for illustration purposes only, really. It's a long-winded way of finding the 'now' date.

$ date -d "1970-01-01 utc + $(date +%s) sec"

Mon Jan 16 10:02:54 GMT 2006

Finally, the same -d option can be used for creating any date stamp, for example:

$ date

Mon Jan 16 10:26:21 GMT 2006

$ date -d "+ 30 min"

Mon Jan 16 10:56:42 GMT 2006

$ date -d "+ 2 days"
Mon Jan 18 10:58:03 GMT 2006

xhost in RHEL4

I needed to illustrate how an operator could send an X application from a server to a workstation in his office (I'm talking here RHEL4 boxes).

Simple, done it loads of times - on the workstation do:

# xhost +server-name

On the server run the application from the command line and direct the output to the display on the workstation:

# xapplication -display workstation-name:0.0

WRONG! It's not that simple on RHEL4! Here, you need to enable TCP first, which is done as part of gdm (graghical desktop manager) configuration.

Run:

# gdmsetup

Select Security tab and notice that the following setting is "checked":

"Always disallow TCP connections to X server (disables all remote connections)"

Take the 'tick' off, restart the GUI (CTRL-ALT-Backspace), and all is well - the method shown above will now work.

I'm still about...

Anybody looking at this blog would be forgiven for thinking that I've given up...
No such luck. I'm still here, except that the project I've been involved in took me into a customer's environment where access to Internet is not easy.

How ironic is that! Internet has clearly become a victim of its own success. I still find it incomprehesible that one might be expected to work without access to e-mail and the web, yet most of large organisations don't seem to provide these facilities to visitor even or even contractors.

Time for 3G or similar?

Polish your Ingres in Polish

Been looking for a book on Ingres, hoping to find something on r3.
Funily enough, couldn't (and din't) find anything on amazon, etc. There are some excellent HOWTOs, notably at www.tldp.org, but nothing recent - most of the documents are several years old. Something more contemporary could prove quite useful. Now, there is a thought ;-)

However, found a book in a Polish on-line technical bookshop!
http://www.ksiegarnia-techniczna.com.pl/modules.php?name=Sklep&plik=lista&nazwa=opis&kod=837101144X

OK, it isn't exactly up-to-date, as was published in 1994, but I still found it amusing that one exists! And at all of 6.50 zl (about £1.00), I decided to order it!

Ingres 3 - part 2

Well, now that the software appears fully installed and configured, the daemons all hovering in the background waiting to spring to life, time to put some data in, and see if one can ever see it again ;-)

Logged in as user ingres, and off we go:

[ingres@fedora4 ~]$ createdb newone
Creating database 'newone' . . .

Creating DBMS System Catalogs . . .
Modifying DBMS System Catalogs . . .
Creating Standard Catalog Interface . . .
Creating Front-end System Catalogs . . .
Creation of database 'newone' completed successfully.

So far so good, now have a go at creating then selecting some data:

[ingres@fedora4 ~]$ sql newone
INGRES TERMINAL MONITOR Copyright 2005 Computer Associates Intl, Inc.
Ingres Linux Version II 3.0.2 (int.lnx/105) login
Fri Oct 21 09:49:07 2005

continue
* create table t1 (col1 char(20));
* insert into t1 values ('hello world');
* select * from t1
* \g
Executing . . .
(1 row)

┌────────────────────┐
│col1 │
├────────────────────┤
│hello world │
└────────────────────┘
(1 row)
continue
* \q
Your SQL statement(s) have been committed.
Ingres Version II 3.0.2 (int.lnx/105) logout
Fri Oct 21 09:52:46 2005

So, what's the conclusion? Well, I'm very impressed.

I suppose I will encounter some problems with the very installation I have just completed, as there are specific environmental settings, as well as kernel parameters that I did not investigate and adjust.

Another important issue a chose to totally ignore is that of user identity when performing the download and installation. This should also have been identified and used correctly. However, what the above exercise proved, is that Ingres for Linux is fine, and can be used 'out of box'.

Basic installation and configuration is simple - if the defaults are fine for you, that is!

Ingres 3 - part 1

A customer of ours is planning an upgrade to Ingres3, whilst changing from Dynix to Linux at the same time. So I thought that I'd refresh my acquaintance with Ingres, since I haven't seen this version yet. I have had some experience of looking after a Linux system running Ingres, but this is the first time I'll be installing r3 myself.

At this stage I'm not even sure that what I'm doing will work. This is going to be what I call a dry run - I'm going to play it by ear, following the installation steps intuitively, rather than following any particular set of instructions - so, the following steps should be taken more as a log of events, rather than a tutorial. I had a look at www.tldp.net , which does have an ingres HOWTO, but it is a really old one, and although most of the document might be OK, I decided not to risk it ;-). At least this way I will have to use some brain cells.

You've been warned! To emphasize the point, here is a quote from the "Ingres r3 Getting Started" document (part of the downloaded documentation package):

"Note: Make sure you thoroughly understand the issues regarding system resources, storage locations, and other configuration parameters before installing Ingres. If you are not sure, have someone more knowledgeable in these areas perform the following tasks."

I have not even checked or tested that system requirements are good enough - here is the briefest of specs:
I'm using Compaq nc6120, with 2G RAM, inside a VMWare virtual machine with Fedora Core 4.

So, to the installation.

Step 1 ================================================================
Download the following files from CA site: http://opensource.ca.com/projects/ingres/

ingres-3.0.2-105-pc-linux-i386.tgz
ingresr3documentation.tar.gz

You will need to register with CA first.

Step 2 ================================================================
Prepare for the installation: create 'infrastructure' and unpack the tarballs.

# mkdir -p ingres/documentation

Unpack the documentation. I chose to separate the documentation (PDF files) from the installation software.
# cp ./ingresr3documentation.tar.gz ingres/documentation
# (cd ingres/documentation; tar xzvf ingresr3documentation.tar.gz)

Unpack the Ingres 3 software
# cd ingres;
# tar xzvf ingres-3.0.2-105-pc-linux-i386.tgz
# cd ingres-3.0.2-105-pc-linux-i386

Step 3 ================================================================
Perform the installation

Need to run the Licence scripts first, to accept the conditions. The main database installation script won't work until you do this one first. Trust me, I know this from experience ;-)

# ./ingres-CATOSL

Can now run the main Ingres 3 installation script.

# ./*install
INSTALL OUTPUT: Invoking RPM...
INSTALL OUTPUT: Preparing... ################################ [100%]
INSTALL OUTPUT: System user ingres has been created but no password has been set.
INSTALL OUTPUT: Please set the password to enable login for this user.
INSTALL OUTPUT: 1:ca-ingres ################################ [ 6%]
INSTALL OUTPUT: II_INSTALLATION configured as II.
INSTALL OUTPUT: 2:ca-ingres-dbms ################################ [ 12%]
INSTALL OUTPUT: Setting up Ingres Intelligent DBMS, this may take some time...
INSTALL OUTPUT: See /opt/CA/IngresII/ingres/files/install.log for more info.
INSTALL OUTPUT: 3:ca-ingres-net ################################ [ 18%]
INSTALL OUTPUT: Building the password validation program 'ingvalidpw'.
INSTALL OUTPUT: Could not compile ingvalidpw: Using executable from the distribution instead.
INSTALL OUTPUT: Executable successfully installed.
INSTALL OUTPUT: 4:ca-ingres-qr_run ################################ [ 24%]
INSTALL OUTPUT: 5:ca-ingres-abf ################################ [ 29%]
INSTALL OUTPUT: 6:ca-ingres-c2audit ################################ [ 35%]
INSTALL OUTPUT: 7:ca-ingres-das ################################ [ 41%]
INSTALL OUTPUT: 8:ca-ingres-esql ################################ [ 47%]
INSTALL OUTPUT: 9:ca-ingres-ice ################################ [ 53%]
INSTALL OUTPUT: Setting up Ingres Web Deployment Option, this may take some time..
INSTALL OUTPUT: See /opt/CA/IngresII/ingres/files/install.log for more info
INSTALL OUTPUT: ngres release 3 has installed successfully.
INSTALL OUTPUT: The instance will now be started...
INSTALL OUTPUT:
INSTALL OUTPUT: Starting Ingres, instance

So, the following Ingres packages have been installed:

# rpm -qa | grep ingres
ca-ingres-net-3.0.2-105
ca-ingres-das-3.0.2-105
ca-ingres-odbc-3.0.2-105
ca-ingres-tuxedo-3.0.2-105
ca-ingres-CATOSL-3.0.2-105
ca-ingres-dbms-3.0.2-105
ca-ingres-qr_run-3.0.2-105
ca-ingres-c2audit-3.0.2-105
ca-ingres-esql-3.0.2-105
ca-ingres-jdbc-3.0.2-105
ca-ingres-ome-3.0.2-105
ca-ingres-star-3.0.2-105
ca-ingres-vision-3.0.2-105
ca-ingres-3.0.2-105
ca-ingres-abf-3.0.2-105
ca-ingres-ice-3.0.2-105
ca-ingres-rep-3.0.2-105
ca-ingres-documentation-3.0.2-105


Service 'rc' file has been installed:

# ls -l /etc/init.d/ingres*
-rwxr-xr-x 1 root root 7187 Oct 20 19:27 /etc/init.d/ingresII

Indeed: the service list in the runlevel 5 lists the following:

Ingres II name server (iigcn) - running
Ingres II recovery server (dmfrcp) - running
Ingres II DBMS server (iidbms) - 1 running
Ingres II Star server (iistar) - 1 running
Ingres II Net server (iigcc) - 1 running
Ingres II Data Access server (iigcd) - 1 running
Ingres II JDBC server (iijdbc) - not active
Ingres II RMCMD process (rmcmd) - running
Ingres II archiver process (dmfacp) - running

And, sure enough, loads of Ingres processes are running:

# ps -ef | grep ingres
ingres 25691 1 0 19:34 pts/1 00:00:00 /opt/CA/IngresII/ingres/bin/iigcn II
ingres 25805 1 0 19:34 pts/1 00:00:02 /opt/CA/IngresII/ingres/bin/iidbms recovery (dmfrcp) II
ingres 25925 1 0 19:34 pts/1 00:00:01 /opt/CA/IngresII/ingres/bin/dmfacp II
ingres 25935 1 0 19:34 pts/1 00:00:19 /opt/CA/IngresII/ingres/bin/iidbms dbms (default) II
ingres 26079 1 0 19:34 pts/1 00:00:00 /opt/CA/IngresII/ingres/bin/iigcc II gcc
ingres 26135 1 0 19:34 pts/1 00:00:00 /opt/CA/IngresII/ingres/bin/iigcd II gcd
ingres 26189 1 0 19:34 pts/1 00:00:02 /opt/CA/IngresII/ingres/bin/iistar star (default) II
ingres 26270 1 0 19:34 pts/1 00:00:02 /opt/CA/IngresII/ingres/bin/rmcmd II

That's all for this posting - next time, I'll see if my common sense approach worked. Chances are that many settings, variables, locations (possibly even additional libraries, etc) will have to be tweaked, adjusted, added...

Why am I so pessimistic? Well, it's just that it's been going too well so far! ;-)

Commenting out a block of lines in a script

Everybody knows of the '#' character as the character allowing to comment out lines that are not meant to be interpreted by the shell in a shell script.

Well, how cumbersome and untidy it ends up sometimes! Imagine you have a block of code, perhaps a big for-do-done loop, or a complex if-elif-else-fi statement and you need to, temporarily, take it out of the script for some reason. Inserting hash in front of each line, and then removing it again when done with whatever test you wanted to conduct is just plain messy.

Ok, you could argue that this can be done nicely in vi :

First, identify line numbers with:

:set nu

Then:

:10,30s/^/# /

will insert the hash at the begining of line, on lines 10 through 30, but even this is long-winded, and if you are using another editor, then the method may not be available at all.

The alternative is to use a combination of the 'do nothing' command: the colon command, and the 'here document'. Imagine the following construct:

: <<>
....
....
....
BLOCK-OF-LINES-COMMENTED-OUT

You can easily move the two lines up and down the script, and isolate the blok of line you don't want to 'participate' at run time.

How does it work? The colon command means 'do nothing', but generate exit status 0, success. The way we are using it here is: 'do nothing, and whilst doing this ;-) use stream of data provided as 'here document'. This embedded data is not not interpreted by the shell at run time, so it hides our code!

The case in date

Tha date command, one of the eldest tools in Unix has got one quirk that escaped me all those years. As usual with these things, it was a delegate who pointed this out to me, with a wry smile.

A bit of background first. If you need to extract hour and minutes from the date command, you'd probably use:

$ date +%H:%M
14:48

The little 'inverted polish logic' (is that why they keep telling me these things? ;-) ) is if you try to use the AM/PM indicator, achieved with the 'p' options.

Try the following:

$ date +%H:%M\ %p
14:49 PM

Then try:

$ date +%H:%M\ %P
14:49 pm

Don't know the background to this, but it does appear a deliberate tease ;-)

Pleasures of a tight-fit hangar

A Pup (150) embracing a Robin (400) 

Still the best kept secret?

I'm amazed how many people are still not aware of what Linux is! And I don't mean any Fred (sorry to all Freds out there!) off the street. I mean IT professionals, who come and join our UNIX classes. OK, so it was a Unix Fundamentals I'm referring to in particular; but even then, I would expect a bit more appreciation of the players in the industry.

We give away a Knoppix disk on our UNIX/Linux courses. Frequently people ask - "what's that?". In itself, this is not an unexpected question, after all Knoppix is not a distribution talked about in board rooms. But when I answer: "it's a distribution of Linux, which...." and I still get a follow up "what's that?", then I do despair!