Blog Host: Robert Crawford - CICS systems programmer READ ENTIRE BIO

Sometimes bureaucracy can be relative benign, if not a little confusing. I worked for one company that changed dataset naming conventions once every couple of years. That wasn't the problem. The problem was that no one cleaned up the old dataset names. This meant that if you wanted to find a dataset you had to remember when it was allocated before you could figure out the name. Since this was before IBM added DSLIST (3.4) to ISPF, I spent many hours hunting and pecking.

Bureaucracy at its worse can be exasperating and demeaning. A friend of mine works in a shop whose first reaction to mistakes is to bite down hard. Most of us who have worked in IT for a while know that sometimes you have a string of problems that have logical explanations, but seem to happen all at once through what can only be called bad luck. Some bosses don't want to hear that. Instead, they choose to put rules around the system and the workers. They forget that, as professionals, we take pride in our ability to keep the systems rolling. They forget that we don't make changes willy-nilly without forethought. They forget that we know it is not in the best interests of our careers to make stupid mistakes.

As bad as bureaucracy can be, sometimes its opposite can be worse. I've noticed a trend towards "self service" in a lot of companies. An example is the HR functions that have been delegated to the employees. Instead of filling out a timesheet that's passed on to a clerk, workers fill in their own pay stubs. It's nice that they trust the employees, but is it really more efficient for thousands of people to do this five minutes work than a handful of clerks doing it full time? And it's even worse for managers who now must do all their HR work and approve the timesheets entered by his or her employees.

If you ever get buried in bureaucracy you should always remember the future will be different. In most companies, the pendulum swings back the other way and everyone breathes a sigh of relief. For others, there's always retirement. Since I'm not quite that old yet I just listen to my MP3 player and try to find my happy place. However, any way you deal with it, you should remember tomorrow just may be better than today.
Posted by Robert Crawford What's your favorite language?
09 FEB 2006 19:25 EST (00:25, GMT)
Programming languages all have their own nuances, quirks and styles. Unfortunately, even though we may find our favorite, the one we ultimately use is dictated by the project we're working on. I'm lucky. Most of the things I write are spot solutions on a small enough scale that I can choose. What follows is an unscientific and bigoted survey of some of my favorite languages.

The first language I learned was PL/1. It served very well to teach newbies like me structured programming and also happens to look a lot like pseudo-code. I still like it even though I use it very rarely. Unfortunately, this PL/1 has been all been crowded out by old (t)rusty COBOL and trendy C++ or Java.

Rexx is a very interesting language. It's flexible and ridiculously easy to write. I also admire its rich set of built-in text processing functions that make Rexx a good choice for parsing reports or filtering messages. Because it's so easily extendable, programmers have added interfaces for things like DB2 and TCP/IP. These extensions make this relatively simple scripting language frighteningly powerful. I'm sure someone out there has written an entire HR system in Rexx. Unfortunately, however wonderful it is, Rexx never really made it off of the mainframe.

The object-oriented languages, Java and C++, are interesting because object-oriented programming (OOP) is a very powerful idea. A programmer can find an existing class, incorporate it into a superclass with a few added methods and attributes and end up with something very useful. Some of the programming is easier, too, because the methods end up being very simple and straightforward. There is a down side. It gets to be expensive to create and destroy objects all the time. We've also had a lot of bad experiences with programmers using a class without really knowing what it does, so we end up with poor performance and bogus database records.

This leads me to Assembler, my language of choice. Not only fast, it comes with a very powerful and flexible macro language. An Assembler programmer doesn't have to worry about a runtime library or how the compiler might interpret the source code. He or she knows exactly what's going on. It also amuses me to be able to divide a character string by four. Lastly, it's about the only time a 46-year-old systems programmer can feel badass. "That's right; it's Assembler," I sneer at lesser mortals.

I'm sure there are lots of languages and features I may have skipped over, but these are the ones I use the most at my job. I sometimes wonder what the next big thing (NBT) is in programming. After structured code and OOP, there has to be something lurking in the universities that's going to make it all obsolete. But I'll always have the last laugh because, before any program can run, it must be translated into Assembler.
Posted by Robert Crawford I'm thinking of a row
08 FEB 2006 13:59 EST (18:59, GMT)
There seem to be a few things that all programmers must do. One of those is search a table for some reference data. There are lots of ways to go through tables, some of which are more efficient than others. In this blog entry I'll talk about some of the methods I've used.

First, of course, there's the sequential search from the top to the bottom. This works well enough for small tables or every-once-in-a-while searches. About the only improvement you can make to this scheme is to sort the table entries by the search data element or key. That way, your search loop ends either when you find your row or a table entry whose key is lower than the search data.

Next comes the binary search. For this to work the table entries must be laid in order in contiguous storage. It's kind of like trying to guess a number between 0 and 100 by asking, "Is it 50?" and then halving the range of numbers at each subsequent try. With this method you use an index to address the row you're interested in. As you go through the loop you must also track the upper and lower bounds of your search. At initialization, the upper bound is the last row in the table while the lower bound is the first. You add the two together and divide by two to get the index of the middle entry. If that's your entry, you're done. If your data is lower, set the higher bound to the current index. If the data is higher, you set the lower bound to the current entry. At the top of the loop you add the bounds together and try again. You know the row isn't in the table when the upper and lower bounds are the same.

The last option is probably the fastest, although it tends to use a lot of storage. It's called hashing and it involves transforming the table's key into an offset of a hash table. The address in a hash table points to entry. The number of entries in the hash table depends on all the possible outcomes from the hashing routine. However, since hashing algorithms aren't perfect, you may have to deal with a "synonym chain," which links together all the table keys that hash to the same offset. As an example, I wrote a program that hashed a four-byte transaction ID by dividing it with the number of table entries. Since I used the remainder from the division as my offset into the hash table, my hash table had as many entries as the main table. Another method uses the PACK instruction to compress the data by eliminating some nibbles (byte halves). With the offset from the hashing routing, you pick up the address of the table entry from the hash table. If the entry matches, you're done. If it didn't, you have to run the synonym chain, if it exists. If you don't find the entry you were looking for at the end of the chain, you know the row wasn't in the table.

Once again, some of these algorithms are easier to implement in some languages than others. The programmer also has to be comfortable with using data as numbers and doing math with pointers. However, there are performance gains especially when performing repetitive searches or looking through long tables.
Posted by Robert Crawford Linked lists
07 FEB 2006 16:08 EST (21:08, GMT)
In my last blog entry, I referred to dynamic structures. In this one, I want to explain how to use linked lists to implement them. Of course, a lot of this was covered in CS 101 so, in the vain assumption anyone is reading, you can skip this one if you've heard it before.

Linked lists are made up of structures or nodes that are linked together by addresses. There are single-linked lists with one chain going forward through the nodes and double-linked lists with two chains, one forward and one backward. Usually a null pointer (or zero) signifies the end of each chain, although there are circular lists where the last node points to the first so there is no end. The linked list itself should have an anchor somewhere in your program that points to the first node.

Linked lists are useful for dynamic structures like queues and stacks. You can also use them to sort incoming data, but that becomes inefficient if the chains become too long.

Searching a linked list isn't difficult as it only involves a looping through the nodes to find what you want. Updating it is a little trickier. For a single-linked list you must put the address of the new node into the pointer of the node to precede it. Then you update the new node's pointer to refer to the one that comes after it. You must also be careful to have exclusive control of the chain during the update if it's in a multi-tasking environment. That may be easily achieved through the use of enqueues, locks or latches.

If you want to avoid the overhead of getting and linking little pieces of memory together, you can use a segmented table. A segmented table is just like a regular one, except it's broken into pieces linked together with addresses. For instance, say you have a list of 16-byte entries you want to put into a segmented table. You get enough storage for eight entries plus a pointer, which totals 132 bytes. You fill up the first eight entries then get the next segment. Use the pointer in the first table segment to point to the next segment and fill the eight entries as before. To search these structures you address the first segment and look through the entries. If you don't find what you want, you address the next segment and scan its entries, and so on. Segmented tables are a good compromise for performance, but are difficult to use if you want to update the list in some collating sequence.

Linked lists are useful devices for maintaining dynamic structures. Of course, programming for them is easier in some languages than others and some people are uncomfortable with pointers. However, link lists can also be implemented a little less efficiently with arrays using indexes instead of pointers. In the end, using them and dynamic structures make programs more flexible and able to deal with varying amounts of information, as well as less likely to break when something breaks a hard-coded limit.
Posted by Robert Crawford Evil genius programming
06 FEB 2006 21:28 EST (02:28, GMT)
A couple years into my career we were joined by a really sharp network guy. He was what you may call a temperamental genius in that he wasn't very personable. He also didn't stay long. It was the early '80s and the job market was very busy. However, he did impress me with a couple of excellent programming rules that have always stuck with me.

• Define things once. If you can get away with it, keep your resource definitions in one place. If there are already some resource definitions you can refer to, use those. For instance, some subsystems like CICS provide a rich API for inquiring on system resources. Use this API if you need to know what's on CICS instead of keeping your own. Nothing looks more like a failure waiting to happen than trying to keep two or more lists in synchronization.

• Use dynamic structures. It's a fact of life that tables get longer and structures get bigger. Write your programs so the size of these things can be changed with little effort and without repetitive editing. This is real easy in Assembler, which has macros and symbolic substitution. If the structures are set up correctly, all you have to do is make your change and recompile. Hard-coded structure lengths and table sizes are time bombs waiting for the poor programmer who makes one change and forgets to make all the others.

  Having said all of that, I would only add that you shouldn't get silly. At one point I maintained a program written by someone else that built a linked list table from an input file. Unfortunately, the number of records in the file was contained in the very first record. I ignorantly added some records to the file, but forgot to update the number of records in the first one. As a result, the records I added didn't show up until I read the source, found out what was going on and updated the record count record.

• Spend a little extra time setting up. The housekeeping section of a program is generally run once and usually consists of a very small slice of the overall processing. Therefore, take the time to set things up so the rest of the program can go through things more quickly. You can rearrange the data or build relationships between the structures, which may greatly simplify your main processing loop.

  Again, you have to apply some common sense. An online program should probably not spend a lot of time on an elaborate setup as there won't be much payback in the rest of the program. However, batch jobs that process for hours might see the benefit.

EXCI calls DFHXCURM every time it needs to open a new "pipe" to a CICS (allocate_pipe command) or during a retryable error. DFHXCURM's parameter list contains, among other things, information about why it was called, the address of the target CICS name (if specified), the user ID making the request and the target program name. When returning from an allocate pipe call DFHXCURM may override the target CICS name. EXCI drives DFHXCURM for retryable errors so the URM can keep track of a region's status.

An important point to remember is that DFHXCURM can control message routing during allocate pipe commands only. Once a pipe is established, DFHXCURM won't be invoked again unless EXCI encounters a retryable error communicating with that CICS.

Also note that before last year an address space could have a maximum of 100 pipes. A PTF (whose number I don't have at hand) increased the limit to 250. If you're hitting the EXCI pipe limit you must increase the number of CTGs. Fortunately, with port sharing you can have several CTGs listening to the same port and your clients won't have to change.

Since CTG calls DFHXCURM by way of EXCI, you may use it to direct where new pipes go. This doesn't work so well in a development system where one CTG may have to talk to several different regions, depending on where the application programmers want to test. In this case, DFHXCURM may only want to keep track of where the pipes are going and write a warning to the console when approaching the EXCI pipe limit.

DFHXCURM makes a better routing tool in production. In production the clients shouldn't specify a target CICS. Instead, you can use DFHXCURM to route the incoming message to an application owning region (AOR) or, my favorite, a listening region (LR). Again, it's important to realize that DFHXCURM won't be called after EXCI establishes the pipe. In production DFHXCURM can send a warning message when it approaches the EXCI pipe limit that may trigger automation by bringing up a new CTG.

It's nice to be able to use DFHXCURM to control pipe allocation, but its abilities are limited. As a future enhancement I'd suggest IBM provide another EXCI exit or URM that would be called for every message going over a pipe. Thus this program could work just like the CICS routing URM's used for remote programs and transactions.
Posted by Robert Crawford Link vs. language calls
02 FEB 2006 19:38 EST (00:38, GMT)
There have recently been some questions regarding which is more efficient; calling another program through native language services or issuing an EXEC CICS LINK command. I can relate from some personal observations that a native language is more efficient, especially if you're using a language requiring a lot of setup, such as C++.

The main problem with using CICS commands is the Language Environment (LE) runtime creates a new enclave for every LINK. Enclave creation is not an easy thing. It involves getting storage for the initial stack and heap allocations (below and above the line), as well as all the block structures. In addition, if you're using C++, there's plenty more to do to build the language environment and initialize objects before any application code runs.

In contrast, a language call avoids enclave creation. The called program's heap and stack are taken out of storage already allocated to the transaction. If there isn't enough room LE getmains more. For C++ most of the work involved in building the language environment is done.

There is a major drawback to using language calls. When you use program calls LE first issues a CICS LOAD for the target program, then branches into it. This logic means the application can't use distributed program link (DPL) and thus binds the transaction to one region. This is not good if you want CICSPlex-capable applications. A compromise solution might be to write code that calls local programs and LINKs to remote ones, but I'm uncomfortable with that because applications shouldn't have to worry that much about their environment.

I guess the upshot is using EXEC CICS LINK commands is fine most of the time. However, you may want to consider language calls when you need a performance boost, are calling a non-CICS program or using a CPU intense language like C++.
Posted by Robert Crawford The mainframe is dead (again)
01 FEB 2006 13:39 EST (18:39, GMT)
Around the middle of each decade since the '80's we start hearing the rumors that the mainframe is dead. Sure, it still processes a lot of data and serves as the back end of some very large companies, but its days are limited. Any day now, CIOs around the world will wake up and chuck the monolithic dinosaurs for something more flexible and cheaper. I know I took the threat seriously enough 10 years ago to take a career detour into writing C++ code on Windows NT.

But, somehow, the big machines manage to survive and end up stronger. IBM has done so well redesigning the hardware that I can't remember the last time we had a processor go down (knock on wood). MVS and other subsystems -- such as CICS, DB2 and IMS -- are exploiting the parallel Sysplex to the benefit of availability and maintenance. And, just in case you need them, we have 64 bits.

There are other things about the mainframe you don't really appreciate until you work for a little while on another platform. After all, how does Unix security stack up against something like RACF? Updating the Windows operating system really makes one appreciate SMP/E, not to mention how difficult it is to safely and completely install or remove software on that platform. And what about these platforms that require upgrades or new boxes when they reach 50% capacity? This is a far cry from a mainframe that barely breaks a sweat when the CPU is in the low 90's.

When I first started in 1981, everyone in IT spoke pretty much the same language. That isn't true anymore, given the complexity of our systems and the choices of platforms. When I venture out of my department, I often find myself explaining to people how mainframes work as if it were some giant mystery. Unfortunately, I'm a small enough person that I've used this to my advantage. For instance, I heard someone talking about some Unix software designed to balance and manage workloads. "Oh, that sounds a lot like WLM, which was based on SRM about 20 years ago," I gleefully pointed out.

Still, as much as we love our behemoths, we have to recognize there are some things they can't do. I've heard statistics that indicate an Intel machine can easily run Java many times faster than a mainframe. Mainframes are also too big and ponderous to be used for spot solutions or small applications. And, to be entirely honest, there are just some things that are better left to smaller machines. Remember the cartoon of the mainframe mouse that was as big as a golf cart?

If mainframes are going to last much longer, some things need to change. I humbly submit my suggestions:

• College-level programs designed to train students on the mainframe. IBM already is pursuing this option.

• Mainframe software costs are outrageous and only get worse as vendors charge for machine or data center capacity instead of actual usage. Software vendors need to back off before they kill the goose.

• IBM needs to reduce the causes and effects of Sysplex "sympathy sickness." These are the cases where one sickly LPAR causes problems for everybody.

• Expand or eliminate limits on the I/O subsystems. Granted, mainframe I/O is very good. However, there are enterprises rubbing against the 64K device limit. Sometime in the future we may also want MVS to hook up to the SAN.

• IBM should keep up the good work started with parallel Sysplex. Someday we may be able to apply and install maintenance in the middle of the week instead of Sunday morning. At my age, I need my sleep.

In the old days CICS programs just had to be quasi-reentrant, which was a strange concept even for CICS. Remember that standard CICS programs all run under a single quasi-reentrant TCB (QR TCB). In addition, because CICS is a cooperative multitasking environment, a program won't lose control of the CPU until it issues a CICS command that incurs a wait or the runaway task timer pops. This meant that a non-reentrant CICS program could successfully multitask as long any changes to program memory were undone before the task lost control of the processor. This seemed to work well for decades, especially when programmers kept their fingers crossed.

However, IBM recognized a weakness in CICS' single TCB structure and set out to address the situation. First off, they created a new TCB structure that isolated some system functions. Then they got rid of the macro API, which also stopped customers from meddling in system control blocks. Next IBM came up with the open TCB's for Java, SSL, sockets and DB2.

Now, when an application makes a DB2 call, CICS dispatches it on one of the DB2 open (L8) TCBs to do the heavy lifting of the database call is done. If control returns to a quasi-reentrant program, CICS redispatches it in the QR TCB. However, if the program is threadsafe, it continues executing on the L8 TCB. The task will stay on the L8 TCB until it links to a quasi-reentrant program or issues a non-threadsafe CICS command.

Running under the open TCB accomplishes a couple of things. First, it allows more work to get done in one CICS address space. Before, everyone had to wait their turn in the QR dispatching chain. Now, each task runs independently and takes its chances with the MVS dispatcher. Second, it avoids some of the pitfalls of making operating system calls from CICS. Since the task is on its own TCB it can issue OS waits, posts or file I/O, although I wouldn't recommend it.

Of course, the really important question is: How can you tell if a program is threadsafe? The answer is there's no easy way. Re-entrancy is a good start, but not an absolute guarantee. There may also be some technique buried deep within the code that violates threadsafe requirements. By way of example, a few weeks ago our programmers were volume testing a supposedly threadsafe application, but couldn't get the rate past three or four transactions per second before the abends started. I looked through the dump and found a parameter list inside of a program instead of dynamic storage. It turns out the threadsafe application called a PL/1 utility that had the parameter list in static memory. Right now they're in the process of recompiling and testing the utility.

IBM has a way to go, too. The CICS Application Programming Reference lists the threadsafe commands. You may notice a lot of your favorites aren't in there. I'm assuming IBM will make the commands threadsafe as they move more and more support into functional domains.

Despite its slow adoption threadsafe programs are the way to go. Not only will one CICS address space be able to do more work, some current application programming restrictions may be lifted. CICS/TS 3.1 takes a further step in that direction by executing threadsafe programs on open TCBs with or without DB2 calls. But, to get there, you have to remain calm, go through your code very carefully, perform stress tests and don't assume you got it right the first time.
Posted by Robert Crawford Why WLM is half of a good idea
30 JAN 2006 09:37 EST (14:37, GMT)
When IBM first announced parallel Sysplex and Workload Manager we figured we would be able to get rid of our performance team in a few years. WLM would take care of workload distribution and the parallel Sysplex would provide the redundancy needed for availability. All we would have to do is occasionally buy more IBM hardware and make the salesman happy.

What we didn't realize was that parallel Sysplex came with the one, big assumption that all the LPARs would be configured the same. I don't know about your shop, but with ever-increasing software costs we find ourselves moving heaven and earth to get the expensive software on the smallest possible processor. We also don't cotton to the idea that we should have to configure each LPAR to handle the entire production workload at one time.

A good case in point is our very CPU-intensive CICS application. For performance and availability reasons we run it on 20 application owning regions (AORs) on four machines. The application's input comes from Transaction Gateways (CTG) clients in WebSphere. Between WebSphere and our CTG server, we use Sysplex Distributor (SD). Sysplex Distributor queries Workload Manager so it can route messages to the machine doing the least amount of work. The CTG servers connect to Listening Regions (LRs) whose job is to route work to the AORs. The problem is that SD, on the advice of WLM, will make very dramatic workload swings. I can tell you I've watched the transaction rate in an LR go from 60 tasks per second to almost nothing because someone submitted a CPU bound batch job.

Now, if we went with IBM's plan for parallel Sysplex we would have to run 20 AORs on each machine, making a total of 80. Instead of going with this unwieldy and inefficient configuration, we cheated. We put three Listening Regions (LRs) on several machines and wrote transaction routing exits to round-robin the workload to the 20 AORs. We control an LPARs workload by moving the AORs around. Thus, if one system gets an upgrade so it can handle more work, we just move more AORs to it.

Granted, IBM has done a lot of things to improve WLM and its associates in the past few years, but there's more room for improvement. Primarily, software that uses WLM information needs to recognize that CICS and other high priority work will get CPU even if the processor appears to be very busy. It would also be nice if we could exercise more control over the "workload balancing." Sometimes, a stupid round-robin algorithm is the best one to use, and we would rather be able to predict where the work is going. Affordable software would be nice, too. If we didn't have to worry so much about some expensive batch reporting system used by three users, we could finally get to the point where all our machines are configured the same.
Posted by Robert Crawford