Lorance Stinson

New BETA AWK CSV Parser

2009-05-31T10:08:00.004-04:00

As Mentioned before on Sunday, December 21, 2008 I've been working on a new version of my AWK CSV parser. Working two jobs and trying to spend time with my Wife consumes an tremendous amount of my time. So unfortunately I have not made very fast progress. I have put up a new BETA version on my website. This still needs more testing, but it is working quite well thus far.

This has many major changes. The function has been renamed and more functions created. The error reporting has changed dramatically. There is also a new function that takes an array and creates a CSV string. I also changed the parser to start array counting at 1. This should put it more in line with how split() works. Please let me know if there are any problems. Either here as a comment or via email @ "LoranceStinson+CSV" at GMAIL dot "The usual".

Please keep in mind this is a BETA version. I have not created a test suite for this yet. Once I get a chance I will work on that. I'm also working to rewrite my CSV Utilities to not only incorporate this new version of the parser but they are also getting some major work done to them as well. I'm changing option handling and will write proper documentation and a complete testing suite.

AWK Golf - pi eddition

2009-01-03T09:04:00.005-05:00

Another one from Stack Overflow. This one to calculate pi using the Leibniz formula for pi. I of course submitted my solution in AWK. By the time I see them, the other languages I know well are already used.

awk 'BEGIN {p=1;for(i=3;i<10^6;i+=4){p=p-1/i+1/(i+2)}print p*4}'
-> 3.14159

More AWK Golf

2008-12-28T19:07:00.004-05:00

An older AWK golf from the Facebook AWK group.

My solution: awk 'NR==1||$0!=p{print;p=$0}'

From the post:

Write an awk script that reads (from file or stdin), but filters out duplicate lines that are adjacent. Input cannot be assumed to be sorted.

e.g.
a
a
a
b
b
c
c
c
a
a

would become:
a
b
c
a

AWK Golf

2008-12-26T16:11:00.008-05:00

Spotted an x-mas themed golf on Stack Overflow. I submitted my solution in AWK.

echo "8" | awk '{s="#";for(i=0;i<$1;i++){printf"%"$1-i"s%s\n","",s;s=s"##"}printf"%"$1"s#\n",""}'

        #
       ###
      #####
     #######
    #########
   ###########
  #############
 ###############
        #

Rewriting my AWK CSV parser

2008-12-21T15:13:00.002-05:00

I posted this over on my personal blog and thought it would be good to copy it here. I know some people who follow this blog will be interested in this.

I've been working on rewriting my AWK CSV Parser for a while now. I'ave made many changes and improvements, like handling whitespace around separators. I've also created a function for creating CSV strings correctly (harder than it seems). With the addition of the new functions the parser has been renamed to csv_parse. Gone is the global error variable, replaced with a function that takes error codes from csv_parse and returns the string equivalent. Today I did more cleanup, fixed more bugs with the changes and the new csv_create function. I've also started incorporating the changes into my CSV Utilities. csv2csv has been the first victim, and I've already fixed a bug I never noticed before. The performance has suffered slightly over the old versions, but I think being correct is worth it.

There is still quite a bit of work to do though. I'm pondering changing to using getopts instead of a simple while; shift; loop in use now. The other csv2 utilities need to be converted over to the new parsing function. The -W option also needs to be added to trimming whitespace around the separators. The testing really needs to be improved. The testing shell script I created and have refined for some of my other projects, like my AWK data libraries (yet to be released) and CharCode, would probably be best.

A big change that I thought would be much harder is a change to newline handling. Now if -1 is passed for the newline string csv_parse returns -1 when a newline was expected. This will allow the caller to handle fetching the required data. Not every instance where csv_parse will be used would read data from the default source. Unfortunately the error codes have had to be rearranged to handle this. With all the other major changes already this didn't seem that big of a deal. And now I finally have a decent base to write a freeze/thaw routine for AWK. I've been pondering writing a routine to freeze arrays to a file and allow later thawing. The new CSV library makes this much easier to create.

These utilities, and the AWK library they are based on, have been a lot of work to get right. They have also been some of the most useful things I have written yet. I have made extensive use of the library and utilities. csv2csv alone is extremely useful for rearranging the columns of CSV data.

Odd AWK

2008-06-21T13:03:00.003-04:00

I've been working on two AWK projects off and on for a while now. I've been meaning to release them, but have not yet finished with them. I hope to do so, with at least one, soon.

One is a 65c02 emulator, complete with assembler, disassembler and even a simple program to combine object files. It's intended to be a teaching tool for both AWK code and 65c02 assembly. It's been on the back burner for a long while now but does actually work. The emulator alone is almost 2,000 lines of AWK, including comments. I have almost all 65c02 instructions emulated, only some strange and seldom used ones are left. The assembler needs a complete rewrite. It rather sucks. Though the expression evaluator makes a nice calculator when removed and put on it's own.

The other is a set of data libraries. There are implementations of singly and doubly linked lists, queues, stacks and binary trees. Each is in it's own file with a complete API. All based on the book Mastering Algorithms with C and the work of Donald E. Knuth. AWK arrays are used to store the actual data. They were only intended to be a learning exercise for myself. I thought others might use them and thus started cleaning them up and adding testing. I'd like to implement a few more interfaces to each and also an array library to get more Perl/C like arrays. AWK arrays are nice, but they are not real arrays. This project will probably be distributed before the 65c02 emulator. And perhaps might even be useful to someone...

AWK CSV Parser Update

2008-04-13T10:09:00.003-04:00

I've made a few improvements to my AWK CSV Parser:

Added a new option to trim spaces from the beginning and end of fields. This option is off by default to match the behavior of Excel and OpenOffice more closely.
Cleaned the code up a bit and added more comments.
Made the return codes indicate the error that occurred. (See the comments in the source)

http://lorance.freeshell.org/csv/

RetroChallenge Winter Warm-up 2008

2007-12-08T13:21:00.000-05:00

The following is the announcement for the RetroChallenge Winter Warm-up 2008. The link to the original announcement is at the bottom. I am running the BBS again this year.

Retrocomputing enthusiasts the World over are cordially invited to participate in the 2008 RetroChallenge Winter Warm-up, commencing January 1, 2008.

RetroChallenge is an opportunity for fans of "obsolete" platforms to spend a month pushing their retrocomputing limits in the company of like-minded geeks. Participants set their own goal(s) and typically keep the community apprised of progress via weblogs or on the Retro BBS.

Challenges are as unique as the individual RetroChallenger -- in the past participants have programmed games; created music and video content; performed dangerous hardware hacks or simply spent a month trying to get an old computer to boot.

RetroChallenge is not so much about competing, as it is about sharing. Think of it as an on-line show-and-tell for RetroGeeks. Because the individual challenges are self-posed, the RetroChallenge is equally well-suited to hoary old wizards and tender newbies alike. It's not about winning, but about *doing*.

RetroChallenge is open to *all* retrocomputing platforms. For the sake of simplicity (and to avoid the usual endless debate), the criteria for what constitutes a "retro system" is left up to the conscience of the participant. Although it's very much a matter of perspective, we would hope that nobody's going to bring their 5-year-old Dell to the games!

The Rules? Well... there really aren't any.
RetroChallenge runs from January 1st, 2008 to January 31st, 2008. Participants are encouraged to announce their intentions prior to December 31st (but stragglers, as always, will be accommodated). Other than that, have fun and try to keep the platform wars to a minimum.

~LD8

Home page: http://retrochallenge.net
Contact : retrochallenge AT ld8 DOT org
Retro BBS: telnet://lorance.no-ip.org
RC Forum : http://68kmla.net/forums/viewforum.php?f=18

Please feel free to post this announcement to appropriate blogs and forums. http://retrochallenge.net/2008/winter/RCWW08PR.TXT

PWMan 0.6

2007-11-12T14:03:00.000-05:00

I've made a few improvements to PWMan:

Added -G for generating random passwords. It can be used with -a, -e and -x.
-x now works with -a and -e.
Organized the source a bit better.

http://lorance.freeshell.org/pwman/

PWMan 0.5

2007-11-10T12:18:00.000-05:00

I've made a few improvements to PWMan:

Spelling and formatting corrections.
Fixed import aborting with no file name when in interactive mode.
Fixed importing. (Problems when importing from STDIN and possible database corruption.)
The password prompt will now always appear when editing an account.
Added a testing script.
Added the -r option to repair a corrupt database.

http://lorance.freeshell.org/pwman/

CharCode

2007-08-11T21:13:00.000-04:00

A Character Code converter and lister. Outputs a table of Binary, Octal, Decimal, Hexadecimal and ASCII representing the input. Input can be a string of characters or numbers in one of the supported bases. Non printable characters are treated as spaces.

http://lorance.freeshell.org/charcode/

PWMan 0.4

2007-08-02T17:08:00.000-04:00

I've made a three small changes to PWMan:

List account name properly when deleting.
Display account and user name when copying a password to the clipboard.
-p alone now acts like -l -p combined when used with an account name.

http://lorance.freeshell.org/pwman/

PWMan 0.3

2007-06-05T22:51:00.000-04:00

I've made a few fixes and an update to PWMan:

Added the option -V to check if a newer version is available. See the manual for details.
Corrected prompt for password when editing entries.
Confirm passwords when adding/editing accounts.
Corrected a typo that prevented notes from being printed when editing accounts.

http://lorance.freeshell.org/pwman/

Rot13

2007-06-01T23:59:00.000-04:00

For fun a while back I wrote some rot13 scripts. I recently updated them. I added a few comments and made them a bit more compact. There are three scripts written in Awk, Shell and Perl.

http://lorance.freeshell.org/rot13/

PWMan

2007-05-24T20:59:00.000-04:00

A password manager written in shell script. The passwords are stored in a flat text file encrypted with OpenSSL.

http://lorance.freeshell.org/pwman/

todo.sh 1.2

2007-05-21T23:07:00.000-04:00

I've made a few fixes to my version of todo.sh. Here is a rundown of what has changed since the previous release.

Removed the remdup command. It is no longer functional.
Changed the help display. The option -h displays help for the command line options. The action help displays the action help.
Cleaned up messages, help and errors.
The edit action no long leaved blanks at the end of the line.
Switched to Perl POD for man pages. The man pages are now generated from the POD source files.

http://lorance.freeshell.org/todo/

TagIT

2007-05-18T22:16:00.000-04:00

I have written a file tagger in Perl. It maintains a list of file and tag associations in a database. Various options are provided to add, modify, delete and search the contents of the database. This should still be considered Beta, though it is quite stable. There are still a few features to implement and it needs some more performance tweaking.

http://lorance.freeshell.org/tagit/

Minor change to todo.sh

2007-05-14T14:26:00.000-04:00

I've made a change to my version of todo.sh. I added the command editc for editing the comment file for a todo.

http://lorance.freeshell.org/todo/

CSV Update

2007-03-01T14:08:00.000-05:00

I have updated my AWK CSV Parser and CSV Utilities to correct a bug found by rattus. It caused the parser to bail when it encountered a newline right after a quote character. The fix is to change the text "pos == length(string)" to "pos >= length(string)". In addition the AWK CSV Parser now has a note that the file is in the public domain and the CSV Utilities has a minor bug fix and an addition.

http://lorance.freeshell.org/csv/ http://lorance.freeshell.org/csvutils/

todo.sh 1.1

2007-02-03T22:02:00.000-05:00

I've made a few fixes to my version of todo.sh and added a few features I've seen people requesting on the group and fixed a few bugs. Here is a rundown of what has changed since the previous release.

Changed how sed is used to make it more portable by eliminating the -i option.
Removed the line "shopt -s extglob" since it only caused errors.
Added the commands listcontexts and listprojects for listing contexts and projects in todo.txt.
Fixed a bug with the history option.
todo.sh will now attempt to create the configuration file and .todo directory if it does not exist.
Terms can be negated when listing todos by prefixing the term with a -.
Made priority coloring simpler. To colorize a priority just add the color variable to the configuration file. No more editing todo.sh to make it work.
Added the command undo for marking completed items as not done.
Added the ability to mark multiple items done at once.
Added the import command. Each line read from STDIN is added as a new todo.
Various other internal modifications.

http://lorance.freeshell.org/todo/

todo.sh

2006-10-22T23:45:00.000-04:00

todo.sh is a very handy task manager. It is written in shell script and stores it's data in plain text files. This has many advantages, too many to go into here. See http://todotxt.com/ for more information. I have made extensive modifications to the original script. The main change involved adding a unique ID to each todo patterned after that used by hiveminder.com. This was done to allow for an activity history and for each todo to have comments. The former because I forget things and have cats, the latter because I tend to have long running tasks. Because the changes require alterations to existing todo.txt, done.txt and .todo files I have included a script to ease migration.

http://lorance.freeshell.org/todo/

CSV Utilities

2006-09-18T21:27:00.000-04:00

I wrote a few utilities for working with CSV file based on my AWK CSV Parser. They were inspired by the posts I keep seeing from people having problems parsing and working with them. They currently include csv2csv, csv2html, csv2sc (experimental) and csv2sql. I am working on another and plan on writing more. Let me know if you have any questions, problems or suggestions. I'll get back to you as soon as I can.

http://lorance.freeshell.org/csvutils/

The NON

2006-09-07T19:02:00.000-04:00

Stephen Jones of SDF has been working on the NON network and held a meeting recently. See the log of the first and second meeting. He has a draft version of the AUP and B y-Laws up. There has been some discussion on the Forum, mostly by smj (Stephen M Jones) and Avo. Early discussion took place in the Bulletin Board (bboard) on SDF. The archives there would be a good place to look for background information. The NON intends to be like the Internet before commercialization. I personally like the idea of the NON. I'm sick of the commercialism and corruption of the Internet. Every where you look are ADs and SPAM. There seems to be no end to it. None of this would have been tolerated on the early Internet and it will not be tolerated on the NON. If you feel the same way why not check it out, visit the forum and contribute. Even better, register for and host a NON site. Right now this will require a static IP and a dedicated system. Hopefully soon a dynamic DNS system will be available. Currently the only hosts I know of are nic.non and sdf.non.

The NON works by creating a separate logical network that exists on top of the Internet. Servers on the NON use the domain .non and do not communicate with non-NON servers. Clients can connect to NON sites by adjusting their DNS. There are three ways to accomplish this. Which ever you choose make backups and/or make notes on your settings. Messing with network settings can be dangerous even if you know what you are doing.

Change your domain to .non and switch your DNS server to 192.94.73.5. This can be done in Unix by changing the file /etc/resolv.conf. This is the simplest and most direct route to the NON. It also completely cuts you off from the rest of the Internet DNS.
Add the DNS server 192.94.73.5 to your search list. This reportedly works but I have not had any success with it.
Add a zone to your DNS server, if you run your own, to catch the .non domain. For my Bind server I added 'zone "non" { type forward; forwarders { 192.94.73.5; }; };' to the named.conf file. This allows all of the systems on my network to access .non hosts without having to mess with their DNS settings.

Apache Log Helper

2006-09-06T21:52:00.000-04:00

This is a script I wrote to help me search and monitor my Apache web server logs. It started as a quick script I wrote at work and quickly grew to a much more useful script in my spare time. I grew weary of typing commands like "grep DBD::Pg /var/log/apache/error.log | tail" over and over. Now it's as simple as "alh -at DBD::Pg". It can search three different logs files, only one at a time, and has many options that make it quite useful. My favorite is the -x option, this runs all output through egrep with a pattern that excludes local IP addresses. All configuration is done with the variables at the top of the script.

One quirk to watch out for is that alh uses the getopt program to parse command line options. getopt is not available on all systems. It also does not handle options with spaces in them. If you need to search with a patter that has spaces I suggest using a period in place of the space or piping through grep for that pattern.

http://lorance.freeshell.org/alh/

Meta: My Scripts

2006-09-06T21:33:00.000-04:00

All of the shell scripts I have written and posted about.

Apache Log Helper - Search and monitor Apache logs more easily.
Awk CSV Parser - Parse CSV files in AWK.
CSV Utilities - Utilities for working with CSV files.
Port/Service Lookup - Lookup ports and services in a services file.
Rot13 - Three scripts for performing rot13.
SHQL - A database written in shell script.
SHQL Caller ID - A Caller ID application that uses SHQL as a database.
todo.sh - A plain text task manager.
TagIT - A file tagger.
Wipefile - Overwrite the contents of files so they can not be recovered.