"Syntax" Highlight a flatfile

Question

I've googled and looked and explored, but it seems most editors configurable syntax highlighting definitions are based on keywords and such. I need one that is based on columns and lines, basically something to highlight different fields in a flat file.

It needs to be more complicated than one line fits all however, there are several different "types" of lines.

A sample is shown below

A9999000055555 333333     55554444422222210102009000000333333
B44444555550000
B44444555550000
B44444555550000
B44444555550000
C9999        0004

Except for "0004" (# of B records) and "10102009" (a date) all other sequential numbers would be actual alpha-numerics representing different user input fields. The first letter is usually designated as a Line type.

The intent of this question is to allow humans to verify the "correctness" of the files format. Typically these files are normally created in an automated way, but business rules change, and old mainframe programs have to be updated (and what ends up happening is not pretty). So the idea is to give those who create these files a way to double-check themselves that the file meets the format. I do not intend for ANY business rule verification here, purely format.

Answer 1

You could look at the RecordEditor RecordEditor it is not a Text editor (it displays the data in a table). It also has single record display + other views available.

The default is to select the record layout but the there is a "prefered layout" which will display each record using the correct layout for the record.

It does allow you to visually check the file

With the RecordEditor you can display the file in a table (following with prefered option

you can also display in a highlighted text view (View >>>> Text View (Highlight Fields))

you can also mix-match, here the current record is also displayed in table format on the right hand side:

---

Alternatively the RecordEditor has macro option. You could

1. Use the built in macros ViewForEachRecordType.js or HeaderDetailFooterTabs.js (missing from some versions) to display each record type on a seperate Tab
2. Write your own (possibly adapted from ViewForEachRecordType.js)

Running ViewForEachRecordType.js

This will generate a seperate Tab for each record type in the File:

---

Another possibility could be Textplorer, I have not tried it myself but it looks like it will highlight fields

Answer 2

You could write a script to add HTML tags to do the coloring. sed (works on lines) and awk (works on columns) might help, especially on linux. Or a perl script. You could use regular expressions to do pattern matching.

You'll also have to add the minimal HTML code to convert it to a HTML file (header and body tags).

Answer 3

Vim! Start with there instructions for changing syntax highlighting and make your way from there. Here's what I think you're saying:

• If a line starts with "A" it will have a series of 12 characters; a space; six chars; some spaces; many chars
• If a line starts with "B" it will have a series 14 chars;
• If a line starts with "C" it will have 4 chars; some spaces; 4 chars

So you can definitely define your highlighting rules based on those requirements.

Answer 4

I asked about what the intended use of such a utility would be and based on a couple keywords ("view and verify before they send") you basically want to do the following:

1. Make sure the file is in standard format with the right number of columns (and rows, maybe)
2. Give some stats like total number of records, total number of items sold, date range represented in file, etc, etc.
3. Save the file to either a central server or to a waiting area for upload later.

The right way to accomplish this, in my opinion, (especially because, as you say, many customers still have mainframes and still get data in formats like this) is to use a custom front-end to get the information about the data to your customer.

The way to do that? You have essentially 2 options: either a) write a file-parser yourself in Java, C#, or C++ (shudder) to give a 'preview' of the data contained in the files. Or b) hire a member or recent graduate (check a portfolio first!) of computer programming classes to code a parser for you. Or the boss' nephew, or your sister's nerdy boyfriend, etc. I don't know what experience or interest you have in such a thing, nor anything about your background, so please forgive me if I'm making undue assumptions. As with so many other things in the world of Technology, you can have any two of fast, cheap, and good.

The lowest cost and fastest option of those presented so far, which span the range of possibilities I'm aware of as a technology professional, would be to bite the bullet, learn some vim (use GVim for Windows) and use syntax highlighting filters there. Second would be to make a series of scripts in something like sed or awk.

Since the input to your problem is essentially a stream of text chopped off at intervals (the line breaks) the processing of the data has to handle input in the same way, and very few programs do so now, because, as you say, this is data generated by an old mainframe program.

Excel might also be of use in importing, but all the lines have to be formatted the same way, so that's still not going to do what you want. One additional thing that comes to mind is that you might be able to use Access to parse such a file and use some VBA to create record counts and compile statistics, but as far as overlaying highlights, that would be less easy. If you know any COBOL developers, this would be a great 1-2 night project for one of them. It might even turn into an open source project on sourceforge.net to free data from the clutches of the mainframe!

Answer 5

I think you're unlikely to find a solution that doesn't involve programming.

Any text editor with syntax-highlighting features more sophisticated than simple keyword matching will necessarily be more complicated, as it is in Vim and Emacs.

For Vim, you can define a highlighting file like this:

syntax clear A B C

syntax match A /^A\%(\a\|\d\)\{13\} \d\{6\} \{5\}\S\+$/
syntax match B /^B\%(\a\|\d\)\{14\}$/
syntax match C /^C\%(\a\|\d\)\{4\} \{8\}\d\{4\}$/

highlight A guifg=darkgreen
highlight B guifg=darkblue
highlight C guifg=royalblue

and load it with :source myhighlight.vim

Then the lines will be highlighted when they're correct, or left unhighlighted if they don't match the regular expression.

It gets more complicated if you want to highlight particular fields within those lines.

Personally, I'd be looking at ways of invoking some script from the editor, which could do all the checking for me. That way, you could use any editor and not be forced to use one which has the highlighting flexibility you need.

Answer 6

I had a similar requirement and I found the solution using one of the samples given in ":sytanx help" screens.

Below is the sample flat file with 2 records of same type. Here record header is "01" which is the first column. After header record there are 4 columns with fixed lenghts given below:

+-----+-----+-----+-----+-----------------+
| No. | Beg | End | Size| Description     |
+-----+-----+-----+-----+-----------------+
|    1|    1|    2|    2|HEADER           |
|    2|    3|    7|    5|Column 2         |
|    2|    8|   11|    4|Column 3         |
|    2|   12|   15|    4|Column 4         |
|    2|   16|   17|    2|Column 5         |
+-----------------------------------------+

Sample records:

01AAAAABBBBCCCCDD
01PPPPPQQQQRRRRSS

" Syntax definition and highlighting for Record 01
:sy region rec1 matchgroup=rec1 start="^01" end="$" contains=r1col1,r1col2,r1col3,r1col4 keepend
:hi rec1 ctermbg=red guibg=red

" Syntax definition and highlighting for next 4 columns from Record 01
:sy region r1col1 matchgroup=r1col1 start="\%3v" end="\%8v" contained
:sy region r1col2 matchgroup=r1col2 start="\%8v" end="\%12v" contained
:sy region r1col3 matchgroup=r1col3 start="\%12v" end="\%16v" contained
:sy region r1col4 matchgroup=r1col4 start="\%16v" end="\%18v" contained

" Highlighted alternate columns with same color
:hi r1col1 ctermbg=green ctermfg=black
:hi r1col2 ctermbg=green guibg=green
:hi r1col3 ctermbg=darkblue ctermfg=white
:hi r1col4 ctermbg=green ctermfg=black

This way we can define "regions" for more records (say Record starts with "02") with different number and widths of columns which may present in the same flatfile but will get highlighted based on the record header.

Answer 7

If I wasn't going to code a front-end for the file, I would probably use Notepad++ as it has a way to define your own language, complete with highlighting.

Download / Main site

All about the user-defined language section

Another example / walkthru

Hopefully these will help you get started.

Answer 8

It looks like your columns are space delimited?

Have you tried just importing them into Excel, and then just applying highlighting/colours to the columns?

If that won't cut it then as others say, you're probably looking at programming. You could probably write a fairly simple Perl or VBScript that'll take that text file and output a syntax highlighted HTML file or similar using something like regexes to match the patterns that you're looking for?

Answer 9

You could try something like Monarch