A little exercise about a recent forum question input field handling in awk and Perl

A little exercise about a recent forum question input field handling in awk and Perl

Just recently the following question was posted to the UNIX scripting group in Linkedin:
remove all duplicate entries in a colon separated list of strings
e.g. a: b:c: b: d:a:a: b:e::f should be transformed to a: b:c: d:e::f

Some of the fields contain spaces which should be preserved in the output of course, there is an empty field too which (to me and other authors) indicates that the fields are not necessarily ordered. Here I wont discuss the suggested solutions, I also did not answer to the original posting because I read it one month too late.

awk
But when reading the question my brain already got working and I could not help to try for myself. The obvious tool of choice for exercises like this is awk because awk has inbuilt mechanisms for viewing lines as a sequence of fields with configurable field separators.

A solution could be

BEGIN {
  FS=":" ;    # field separator
  ORS=":"     # output record separator
}
{ for(i=1;i<=NF;i++) {  # for all input fields
    if( f[$i] ) {       # check if array entry for field already exists
     continue;          # if yes: go to next field
    } else {
     print $i;          # if no: print the field content
     f[$i] = 1;          # and record it in array f    }  }
}

which leads to this output:
a: b:c: d:e::f:

The script can be shortened by omitting superfluous braces and else to

BEGIN { FS=":" ; ORS=":" } 
{ for(i=1;i<=NF;i++) { if(f[$i]) continue; f[$i]=1; print $i; } } 

The script uses a very simple straightforward logic: loop through all input fields, if a field is new then print it, if not skip it. This is achieved by storing each field in an associated array f when it first occurs.
Using the field separator FS for splitting the input line and the output record separator ORS when printing (you need to know that print automatically adds ORS) makes this an easy task.

There is one issue though: this solution adds an extra colon at the very end (compared to the requested output), this could be an issue or not depending on the context of this request so one might prefer this code:

BEGIN { FS=":" } 
{ printf $1; f[$1]=1; 
  for(i=2;i<=NF;i++) { if(f[$i]) continue; f[$i]=1; printf FS $i } }

which uses a slightly different logic: the first field is printed straight away (and recorded), the loop checks the remaining fields 2..NF and prints the field separator as a prefix to the field content. This code also works for the extreme case where there is just one field and no colon.

Perl
I then wondered if this couldnt be done equivalently or even shorter in Perl but my best solution is a little bit lengthier because I have to use split to get the individual fields.

$FS=":";
@s = split($FS,<>);
for($i=0;$i<=$#s;$i++) {$e=$s[$i]; next if(exists($f{$e})); $f{$e}=1; print $e,$FS }

I could have used command line options "-a -F:" to avoid the split but I need FS to be defined anyway for the output (I dont know if the split pattern defined by -F can be accessed in Perl).
I use split to chop up the input line and put it into an array s. Then the same logic applies as in awk. Instead of an associative array Im using a hash table f in Perl. The variable e is only used to avoid repeated occurances of $s[$i]. In the end tits a matter of personal preference which solution you take.


It should be noted that I tested with

echo -n "...." | awk ... or perl -e ...

which feeds a string without newline to the pipe which helped to avoid chomp in Perl for removing the newline in the last field.

Read More

A general approach to command line switches and their default values in Perl

A general approach to command line switches and their default values in Perl

In the UNIX world youll rarely find a program which doesnt support a few or many arguments (or command line parameters) which influence the execution of the program.

When Perl programs require arguments (one of the simplest cases: an input filename) one could investigate the ARGV hash (an approach which works well in easy cases) or one could turn to one of the Perl modules, in particular if the arguments are command line switches.

In this article I will discuss a few types of command line switches and the possible logic behind.

What is a command line switch?

Just to recap: a command line switch is traditionally denoted as a hyphen followed by a letter optionally followed by a value e.g.  -d or  -d 25 . Note the space between the switch and its value. Some programs require this space whereas other require the value to be attached to the switch like -d25 and still others allow both. Some programs allow switches to be concatenated like  -ltr instead of  -l -t -r . Others allow switches to be more than one letter. Some programs allow a switch to appear multiple times like -v in awk.
Further complexities exist: one switch might override others. Some switches exclude each other mutually.
All these cases would need to be handled properly.

On top of that (I think it was) the GNU world introduced double hyphen switches with (usually) string switches e.g.  --verbose .

In the remainder of this article I will only use the simple case of single letter switches with or without argument. I will be using Getopt::Std, one of the core Perl modules and its function getopts. Its basic usage is  getopts(ab:,%opts); for two switches  -a and  -b foo.

Various types of command line switches with or without default values

The typical distinction between command line switches is whether they are boolean (switched on or off) or carry an additional argument. Then there is the question: if a command line switch is absent should there be a default value used in the program?

The following table explains the differences and shows a few examples.

switch	Example	Default	Notes
-a	...	false	A boolean switch by its very nature has a default value true or false which should be the opposite of what the switch intends to trigger.
-d $HOME/tmp	output directory	/tmp	Certain things in the program require a default value e.g. the program needs to know where to store its output files. Its left to the programmer to decide which of the default values can be overruled by command line switches.
-u joe,sandy	user list	current user	Some command line switches can take more complex arguments, in this case a comma separated list of users. Its absence should be covered by a reasonable default value e.g. the current user.
-p 1507	process id	all processes	Some switches do specify a setting which acts as a filter or a kind of a restriction but its absence does not imply a default value but is somewhat vague. In the user list example before another default behaviour could have been all users instead of current user.

Rather than defining a list of variables to set the defaults like

 $OUTDIR = "/tmp"; $USERS = $ENV{USER}; ... 

and later somehow associate these variables with the switches a (in my view) cleaner approach is to

define the defaults in a hash (the keys are the switches)

create a new hash (again with switches for keys) and set them to either the defaults or values supplied by the command line
The following Perl program handles the cases above.

boolean switches and unspecified defaults are set to undef, all others are set to their reasonable default values.

the  ... ? ... : ... operator is used to set the actual variables
(Getopt::Std sets boolean switches to 1 which represents true, the opposite (and default) could be anything that evaluates to false in an if(...) clause, I chose undef rather than 0).

 #!/usr/bin/perl use strict; use Getopt::Std; # to process command line arguments # Define the defaults in a hash my %defaults; $defaults{"a"} = undef; $defaults{"d"} = "/tmp"; $defaults{"u"} = $ENV{USER}; $defaults{"p"} = undef; # Retrieve the command line switches into a hash # making sure which ones are boolean and which require an argument with : my %opts; getopts(ad:u:p:,%opts); # Put either the default values or the command line switch arguments into a hash my %vars; foreach my $key (keys %defaults) { $vars{$key} = exists $opts{$key} ? $opts{$key} : $defaults{$key} ; } # Test output: see what is contained in vars foreach my $key (keys %vars) { print $key," ",$vars{$key}," "; } print " "; # Check decision tree for boolean and unspecified switches print "a is set " if( $vars{"a"} ); print "p: all processes " unless( $vars{"p"} ); 

If run without any command line switches:

 u andreas p a d /tmp p: all processes 

With -a and -u

 ... -a -u joe,sandy u joe,sandy p a 1 d /tmp a is set p: all processes 

With -p and -d

 ... -d $HOME/tmp -p 1507 u andreas p 1507 a d /export/home/andreas/tmp 

With this general approach one hash vars contains all the information and its contents can be used directly later in the program (like -d output directory) or used in a decision process defined vs. undefined.

Of course there are more issues like the ones mentioned above (e.g. conflicting switches) or validity of values (e.g. does the output directory exist and is writable) but they need to be resolved somewhere else in the code.

Read More