Over the years I have collected a few mementos of projects I have worked on. Some are product information documents, others are various thank-yous I have received from my managers or peers along the way. Following are a few I got around to scanning in.

Let me apologize for the (dis)organization of this page, as I am adding prose as I find documents to scan. As time goes on the quality should improve.

Web-Based Configuration Management and Build System

Most recently I have been consulting independently. I built a web-based build and software configuration manager for one client. The software product was an embedded Linux test instrument that I cannot discuss due to my intellectual property agreement. This system started with a basic Linux system that then needed to be configured with different device drivers and algorithms depending on the customer needs.

The salesman would access a series of web forms that would allow him to select what options the customer wanted. Rules prevented designing an illegal configuration. The customer information and the configuration design were stored in databases.

A build technician would use build screens to find the pending orders. The build system ran under Apache. Pre-compiled components were merged with components that needed to be created on demand. The result was a Compact Disc image directory that could then be loaded onto the targets system. Source code as well as customer documents were automatically stored to and retrieved from CVS on Linux as needed.

Test Instrument Database Manager

Another client, OpQua, Incorporated, is developing a hand-held test instrument for reading configuration information stored in the EEPROMs in pluggable optical modules such as SFP and XFP modules. Their instrument has a local display that allows a technician to review the contents of the EEPROM including the device manufacturer, model number, serial number, and operating parameters.

The module reader has a serial interface that allows it to be used in a stockroom application where the data output could be stored in a database. My assignment was to create an application that could be used as a suitable for small users, and a proof of concept demonstration for larger customers.

The result was a form-based GUI application supported by and SQL database. The GUI was implemented with Perl/Tk. Perl was chosen since there are a variety of modules available that can be integrated for the various interfaces required for the project and the result can be ported easily between Microsoft Windows (the expected initial execution platform), Linux, and Unix.

The Perl Win32::SerialPort module provided access to the Microsoft Windows serial port drivers. The Tk Dialog, ROText, Scrollbar, Table and other modules provided for screen formatting. The Win32::ODBC module provided a portable connection to the database subsystem using the Microsoft Windows ODBC32 service. MySql was chosen as the primary database since it is available on all anticipated platforms and it supports ODBC. The application also works with the Microsoft Access database through ODBC, but MySql was preferred due to licensing costs.

Web-Based SNMP Network Element Manager

One of the products that I developed at Jedai Broadband Networks was a mini-element manager system for the Jedai FrontRunner 3200 product. The FrontRunner was an access device for optical networks. It has both 10/100BaseT (and the fiber variants) packet data and T1/E1 TDM interfaces on the service side and Gigabit Ethernet ring interfaces on the network side. There could be up to 32 customer side interfaces on the 3200.

The manager was to provide two purposes: it was to support enhanced MIB browsing and also provide calculations and generate provisioning information for proprietary features within the Jedai network.

The design used CGI programs written in Bash (and Korn) shell and C for user interface. These and straight HTML files were served under Apache. Information about the network and the elements was stored in Unix-style databases. When used on the PC platform, the Cygwin package provided the shell required. Network maps and other graphics were created on the fly with the Dot package.

The SNMP drivers were provided by the Net-SNMP package. The proprietary MIBs for the FrontRunner use the Interniche Technologies MIB compiler. This compiler generated C source files used in the FrontRunner 3200 application. I wrote a series of Bash Shell and AwK scripts as well as C programs to convert the Interniche output to a set of machine-generated source files that could be used for a fast web lookup agent. This was essentially a compiled MIB cross-reference engine. To test the engine I created a very simple Command-Line operated MIB browser that used the Net-SNMP libraries for network access. The web version of this tool generated SNMP requests based upon commands passed from controlling web pages through either the environment (QUERY_STRING) or from stdin (Post data), initiated volleys of SNMP requests through Net-SNMP, cooked the responses and generated formatted HTML displays of the result.

A key feature of the system was to maintain knowledge of the network topology and to make provisioning decisions based upon requests for new service installations. This required analysis of the system interconnection map to determine best routes for services to take, and to determine how end and intermediate nodes need to be provisioned.

Terminal Server

To support our lab environment, we needed to be able to communicate with the console ports of the units in the lab from our offices. I created a simple application that would allow you to all serial ports on a Windows computer. We then put a computer in the lab with multiple serial port cards and added my server. The server listened on a particular Socket for remote connection requests from applications such as Hyperterm or Xterm. The application would determine what serial ports were available at the time, and display a table on the caller's terminal. The caller could then select a port and speed. The application would then fork a co-process to serve communications between the serving computer and the clients.

Boot and Control of SONET Multiplexers

The initial Jedai alpha products used a SONET backbone. Three products were defined: an OC12 to 100BASE-T edge device, and OC12 to OC3 edge device, and and OC48 to OC12 core device. The first was designed and built by a partner company and they were responsible for providing the operational code. The second was designed by an outside company, but we were responsible for developing the code. The third was designed entirely inside the company.

I took responsibility for delivering the products to the alpha trial per the requirements of our business plan. I studied the hardware for the last two products to determine what code was required for operation in the alpha trial. Since we had yet to hire programming staff, I had to design and plan in such a way as to handle unknown human resources. I designed a system that booted from flash, could store operational parameters in a simple flash file system, and allowed setting and saving of SONET cross connect assignments. I guided and trained new employees as they were added to the project.

These devices use the Motorola PowerPC 860 processor. Initial functionality did not require an operating system, though later on in the project as time permitted a VxWorks BSP was created and the application was ported to VxWorks.

Configuration Management, Source Code Control, Build Management

Rolling off the alpha assignment, I took the role of building staff skills as the software development department grew. In order to help the team work well together, I implemented and deployed software tools. Not all of the development was done with the VxWorks IDE, several different code editors were used. As a common denominator, the Cygwin tools were deployed to all PCs. This gave us a Linux/Unix style fallback for tool management.

A Windows 2000 Professional PC was configured as a CVS file server. Developers used either WinCVS or the Cygwin command line clients for connecting remotely to the server. Since the CVS repository files were not exposed to the network, the were protected from accidental damage by browsers.

There were several projects running in parallel. The major projects used quite a bit of purchased code (several million lines). The CVS repositories had to be carefully designed to support different maintenance rules for purchased versus locally designed code as well as to support modular portability from project to project.

To make it easier for developers of varying skill levels to keep track of changes to code bases and versions an Apache server was added to the CVS server PC. Web pages were added that allowed developers to easily access change logs and to see who was working on what. I designed an automatic build system that was triggered whenever new code was checked in. It also automatically tagged an build code nightly so we could track our progress. Only properly tagged and build versions were passed to the system test organization.

To keep track of bugs and fixes, a Linux server was added that ran Bugzilla under Apache. Developers and system testers could enter bug reports from the web interface and team leaders and managers could dispatch assignments as well. I developed a bridge between the Bugzilla bug database and the CVS change database that annotated bug reports with lists of files and commit comments associated with the bug fix. It was also possible to prevent code commits if the developer did not have an appropriately assigned bug.

I received a Peer-to-Peer award from Jedai. I value Peer-to-Peer awards greatly since in the final analysis, if you can't get along with and support your peers, you cannot succeed, and your project and product cannot succeed.

SoftModem Re-Architecture for Software Manufacturability

Prior to working at Jedai I was a consultant through Tropaion. My last project through Tropaion was for the Elemedia organization of Lucent Technologies. The project was to produce a working V.90 modem with the source code primarily written in the C language instead of DSP assembly language. The intention was to allow the product to be easily ported between processors and operating systems.

As I entered the project, the basic architecture of the code existed. It was based on the C algorithm simulations created during the development of DSP-based modems. Unfortunately, developed in that way meant several different coding and debug methods were used. The code was tested with a loopback simulation on a Windows PC. There were two simulations supported. One had a C++ GUI which displayed constellation diagrams as the modem ran. Another simulation ran from the command line and just produced data for analysis. A port to VxWorks on PC hardware had just started and the controller portion of the modem was being ported to the data pump.

In order to be profitable, the modem had to be very easy to implement and install by our customers. They should not get the modem as source code since that would require a lot of work on their part to understand, build, and test the code and would cost us a lot to support their efforts. In addition, it would be difficult to allow a customer to simply try it out since we would be exposing all of our source code to them.

To simplify delivery I decided to change this to a binary product much in the way the pSOS was delivered. The binary was simply loaded into memory on the target and the customer's application would just jump through know locations in the module to perform the interface functions. A simple C example program would be used as the model to do their own ports. The result would be a compiler and operating system independent module. It would cost us very little to support. We would port from processor type to processor type, but we would not have to worry too much about the customer's operating system and tool choices.

Unfortunately, the organization of the code made that a long term goal. There were too many external references. We made an interim goal of producing binary library product instead. This was also shipped with example code that the customers could build their applications from. The downside is that we would have to build a different library package for each compiler choice for each processor.

Having made that decision, the code needed to be reorganized. The code had a lot of static structures that affected re-entrancy. These had to be identified and the code around them needed to be re-structured as well. The interfaces to the application had to be simplified and formalized. Calls to 'standard' libraries had to be eliminated where possible since not all compiler libraries will provide all the functions that were in use, or they themselves may have re-entrancy problems that would not meet the requirements of our product.

By the time the project wound down, we had restructured the code per my specification and ported it to work for the VxWorks architecture and tools. Libraries were created that supported the Pentium, and the Hitachi SH3-DSP and SH4 processors. I had created a build system that allowed simultaneous creation of all of the target libraries and distribution CDs. I designed and documented the reference designs so that customers could port them to proprietary targets in a matter of days.

Of course, the first customer not only did not use VxWorks, but the did not use a compiler tool chain compatible with our libraries for the SH3-DSP. As a result I had to make a two-prong attack to keep the customer happy. First, I had to port the code and build procedures to use the Hitachi compiler tools and to produce equivalent libraries. This was difficult because code that function properly with one tool chain did not work with the other. Optimizations that ran fast with one did not necessarily work with the other.

The second prong of the attack was to create the relocatable binary as I had originally intended. This was built from the standard SH3-DSP tool chain used for our VxWorks targets. I took one of the standard demo applications and converted it to change our function-call interface to a jump-table interface. The result was compiled and linked with no external references. An additions C file added to the old sample applications converted the jump-table interface back to our published API. After this was completed we were able to drop the binary in our first customer's target, under their operating system and execute flawlessly.

Real-Time Performance Analysis

A modem requires a lot of computations for operation. It is very difficult for a modem written substantially in C to execute within the limits of conventional embedded microprocessors (available at the time). There are two types of problems you could encounter: first, the program could simply be too fat and will never work with a CPU at the given speed. For instance, the code may require a constant 200MIPs, but the processor only provides 100MIPS.

The second problem is more subtle and is related to the sample-driven nature of the application. The hardware does analog to digital conversions at a predetermined rate. This produces a stream of samples to the modem input. The modem analyzes the input sample stream and calculates new digital values to be sent out the phone line. In a simple case the hardware would interrupt the application once each time a sample was available. The modem would be in the interrupt service routine. The modem would crunch the input sample, and then write the output sample to the hardware, and then return the CPU to the application. If the modem took too much time processing a sample, it would not produce the output sample before it was needed. This would distort the transmitted signal, and probably would terminate the data call.

The modem is a state machine. Different states require different numbers of MIPs. It is quite possible (as we discovered the hard way) for the modem to require and average of say 50MIPs as measured in a loopback simulation, but it would still crash on the real target. Most of the time the modem had plenty of time to complete its tasks during a sample period, but occasionally it didn't. This type of problem was very hard to find analytically. Initially, people went around 'speeding things up', but in many cases it was a wasted effort.

To get a better handle on the problem, I ported the loopback simulation (that ran under Windows) to one of our targets. I made the whole thing run as a single task. Next, I took advantage of the way the GNU compiler (that comes with VxWorks) operates. It generates assembly language code from the C code and passes it to the assembler. I modified the makefiles so I could selectively stop compilation prior to assembly. I then wrote a collection of shell and C programs to process the assembly language, These tools could find the starts and ends of C functions. The tools added a few lines of assembly language that wrote a tag number and a processor cycle count to a software trace buffer. The resulting assembly language was assembled and linked into the loopback simulation.

As the modem ran, each time an instrumented function was called or returned a queue entry was created. The modem was allowed to start up and get into the normal operating mode. The simulation was stopped, and the trace buffer was transferred to a PC. Post-processing tools were used to generate a report that gave us sample-by-sample reports of how many CPU cycles were required to process each sample. When plotted with Excel, we found that very deterministic instances where the modem required much more CPU cycles to complete a task than the processor could provide. In some cases these CPU spikes were in the thousands of MIPs. By inspecting the function-by-function CPU utilization statistics for that block, the MIPS usage could be easily identified.

As a result, problems that were unsolved for weeks, were identified and surgically repaired in days in most cases.

print CGI=HASH(0x812c000)->h4( SoftModem "Agent" Development

As I mentioned before, we provided sample applications on several different boards with the different supported processors. These we referred to as Agents. For each of these I wrote application notes with the theory of operation for the particular example. This went along with our API document to aid them in their designs.

Most of the agents were pretty obvious, the modem used a standard telephone interface on one side and a serial interface on the other side. A typical real application would have the application on the same board as the modem so the serial interface was a distraction. Most of the boards had Ethernet interfaces, so we build applications where the serial interface was replaced with a socket interface. We could then connect to the modem with HyperTerm or a data load generator and send data faster than would be possible with the serial interface.

CMDA Cell Phone

Prior to my SoftModem contract, I was a lead software developer on a CDMA Cell Phone project at a division of Lucent that ended up partnering with Philips. I was responsible for:

• the decomposition of purchased software modules
• integration of those module with in-house modules
• modification of the code from IS95 to TSB74
• bringing up the system on the operating system (RTXC)
• supporting peer developers with build and configuration management tools
• design an implementation of performance analysis instrumentation and tools
• support of system and field test operations
• training of, and tasking new recruits
• participating in the re-architecture from an Intel 80C188 design to an ARM/Thumb design

The organization was created from several groups within Bell Laboratories a few months before. Many of the people came directly the research areas of the labs and so we had top notch experts in modulation, demodulation, and RF hardware design. A great robotic manufacturing facility was created.

The Omaha project project was seeded from an older dual-mode AMPS/CDMA phone design that was partially completed. This phone was being designed in partnership with a Japanese company which provided software for the AMPs and IS95 CDMA Call Processing. The DSP code for the CDMA modem was under development by the Bell Labs engineers as was the user interface.

I was brought in as my previous Lucent contract was completing due to my experience with the implementation of an IS95/TSB74 CDMA modem and the associated over-the-air messaging for call processing. I had experience with both the Lucent cell site operation and that of the Qualcomm base station modem and cellular phones. A Lucent employee with similar skills was also transferred to the project at the same time.

The mission was to complete the port of the Call Processing code, integrate it with the DSP physical layer, add Over The Air Service Provisioning (OTASP) and add the user interface in about three months. This port required upgrade from IS95 to TSB74 as well as to move from one compiler tool chain to another. Unfortunately, there were no software design documents written in English, so the code needed to be reverse engineered.

The software ran on and Intel 80C188 microprocessor and used the RTXC operating system. The Paradigm Checkmate debugger and emulator were used for initial debugging of the system. The software was compiled with the Borland C++ compiler version 4.5 on Windows 95 computers. The AT&T Sablime (a.k.a. Sable) source code control system was used on a Sun server. Since we had a mixed Windows/Unix environment, I set up a coding and build environment using the MKS Toolkit which provides ports of most of the common Unix software tools. The build scripts I wrote implemented a viewpath build method. This allowed the product to be built from multiple source code trees with the same layout. That way individuals could work together on changes with out having to have duplicate sets of hundreds of files. The scripts were made simple enough so that only four steps were required to execute a build. A new programmer could be productive on the first day.

There were a great deal of problems getting to the original code to compile. For the first several months I spent segregating the layers of the code and building it from bottom to top. As a layer was completed, it could be added to the base and handed to the my partner in crime in the lab for testing and debugging. It was necessary to create separate execution areas for the call processing/hardware integration and user interface teams since the API between the original Japanese design and the new user interface was discarded and so much of the work on one side would interfere with the work on the other side.

Cell Phone Design Analysis Tools

The original software architecture had no provision for software debugging. The emulators and hardware debuggers were of limited use, unless the code out and out crashed, since hitting a breakpoint or viewing memory would steal enough CPU cycles to crash a call in progress. It was difficult to separate software bugs from interference from the emulator. I created a software approach to debugging, This was designed carefully since if it added too much execution overhead it would interfere with real time and if it was too big we would not be able to fit it into memory and so could not use it in a closed-up phone.

I developed a set of very efficient tracing functions that could be used anywhere in the code. Trace points were organized into channels and messages. Entire channels could be disabled with a bitmask. Likewise, messages within a channel could be disabled. Typically, no ASCII messages were used or needed. The trace functions were passed the address and length of a buffer containing useful trace information. If tracing was turned off the data was never copied. Typically the information to be traced was a data structure.

The tracing functions would copy enabled messages into a circular octet queue. A serial serial manager task would accept read requests from a PC and DMA up to 4 kilobytes of records at a time out of the phone at 56 kilobits per second. A differential timestamp mechanism allowed each message's queue arrival time to be tracked down to sub-millisecond accuracy synchronized to CDMA time and as a result to GPS time. It was possible to compare logs from multiple phones in the same area to see how one interfered with the other.

A PC would log the raw output from the phone to files. A post-processing program named ASLPAR (A Simple Little Parser) was created that read the log files, regenerated system time and could dump the logged records. For some messages it was sufficient to just see the channel number and message number. For others, message cooking was added for a more understandable report. The modular design of ASLPAR allowed it to be maintained in parallel by several people working on different parts of the project. The compiler we used for the phone was also used for ASLPAR and this allowed us to share header files with the phone code. This made it easy to cast a buffer to a structure and provide very informative cooked messages when a trace passed a raw structure. A good example of this was the real-time logging of the over the air messages on the Sync, Paging, Access, and Traffic channels. The message queue interface between the DSP and the 80188 was also traced so it was easy to see if data was delivered to the physical layer with proper timing.

Performance Analysis Tools

Once the phone had basic functionality, we needed to evaluate basic performance. For instance, it was necessary to understand how pilots were managed, was the phone sensitive to find nearby cell sites as the phone was driven around? Was the idle hand-off mechanism sufficiently tuned so that it did not hand off too quickly or wait too long? To verify call processing the phone was used on drive tests where a set of phones was driven around a 50 or so mile track on a live system. A calls test repeatedly made call attempts to verify that the phone could get to the traffic channel, and in a timely manner. A call hold test started a call and let it stay up as long as possible. This was to test the phone's ability to track the CDMA signal and hand-off as the call progressed. The logging feature built into the phone was left on during these test to capture as much information as possible. If a call failed, the logs were the only tool we had to discover the cause.

A single phone might produce 50 megabytes of binary data per day, and a dozen or so phones were used at once. This was a lot of data, especially for the late 1990's. ASLPAR typically expanded that data to about 20 or 30 times that size. This was far to much text to page through looking for call drops. For any useful analysis, additional tools were required.

The record nature of the phone log made it easy to design custom programs to look for specific information from a log file. A typical tool would look for all the pilot measurement reports. With recovered time, the pilot energies could be put into a comma-separated (CSV) file that could be read into Microsoft Excel. You could then see pilot values rising and falling as the phone moved and you could see if hand-offs were executed at correct pilot strengths.

Most of the tools correlated many different seemingly unrelated events in the phone. For instance you could compare the amplitude of Access Probes to pilot energy. You could could confirm that the phone displayed the correct error message on the display after a particular call failure.

CDMA Base Station

I worked for AT&T (as it turned into Lucent) in Whippany, I worked on their CDMA feature of the Autoplex Series II cellular base station. First, I worked on the joint development of the 13kbps CDMA Modem as AT&T's representative with Qualcomm in Boulder, Colorado. This interesting project included moving the 8kbps reference from an old C compiler to a C++ compiler, from Qualcomm's in-house operating system to PSOS, and from a DOS-based development platform to a Unix-based platform. I was there to act as a liason between the Qualcomm and Bell Labs developers and pre-integrate the modem to the new Bell Labs hardware. The development testing of the modem was executed in a mock up system.

In order to make the eventual move of the Qualcomm code into the Bell Labs coding environment I instituted changes in the way Qualcomm developed that code, at least for us. The original 8KBps modem was developed on a mix of Windows and Macintosh PCs and used an Intel compiler. Bell Labs used Sun workstations and had adopted a Microtec compiler and Xray debugging system since they were migrating to the pSOS operating system.

I deployed Sun workstations to the Qualcomm team. I trained them, as necessary, on the differences between developing software on Unix and the MSDOS environment (what a pain it was to fix all of the mixed-case names of include files!!). In addition, I needed to get them to use source code control. The initial goal was to use the AT&T Sablime source code control system like the Bell Labs team was using, but this turned out to not be a timely solution. Instead, I set up GNU RCS and wrote appropriate scripts to snapshot revisions.

I paired with the initial Qualcomm team member in the effort to move the old 8K modem code to the new tools. We decided to take baby steps so we could test along the way. The first step was to switch the makefiles from the Intel compiler to the Microtec C compiler. We discovered a lot of C syntactical conventions that the Microtec C compiler did not support, Also, the Microtec compiler had different bugs than the Intel compiler so some recoding was required to get proper code to be generated.

Once accomplishing our first baselined build, we executed the Qualcomm regression tests for the modem to confirm that the modem would still make calls. Next, we switched to the Microtec C++ compiler that was the compiler of choice for Bell Labs. More coding issues arose since the original C code was not ANSII-conforming. Again we ran into a different set of compiler bugs which required some tweaking of the code. Again we re-ran the regression tests to verify that the modem still functioned properly.

The last port of the 8K modem was from the Qualcomm-proprietary REX operating system to pSOS which Bell Labs required. Bell Labs provided an abstraction of the pSOS API that the modem had to be ported to. The execution and communications models were quite different between the two operating systems, so a lot of code changes were required. We were able to excise REX and get the modem to execute in a few modes. However, the 8K modem design ticked at the CDMA symbol rate (208 microseconds) which was too fast for pSOS, even with an Intel i960 processor. Therefore, we were not able to make phone calls during regression testing nor confirm that the logical port the the new operating system was correct.

Next the new code base for the 8K modem had to be restructured to work with the new 13Kbps CSM (Cell Site Modem) integrated circuit Qualcomm had developed. The new chip handled a lot of the lower physical layer functionality, in particular soft-symbol processing, that used to be done in software in the older modem. This changed the clock tick from the 208 microsecond symbol rate to the easier 1.5 milliseconds power-control-group rate. The corresponding lower layers of the software needed to be removed, and new drivers for the new chip had to be added. When this process was complete, we had a working 8kbps CDMA modem running on pSOS on the new hardware.

The next step was to modify the modem to support 13Kbps TSB-74 PCS mode. Most of this from the modem perspective was augmenting some existing message decoding and formatting and adding new messages. Various state machines also had to be updated. In order to drive the modem, the Qualcomm simulated base station also had to be upgraded to support the new call processing messaging. In the end, we had a modem that quite successfully made both 8K and 13K calls on the new Bell Labs hardware.

Autoplex Series II CDMA Cluster Integration

Once the modem was complete, I brought that back to Whippany and proceeded to build the software platform for the remaining code on the CDMA Channel Unit (CCU) which contained the modem. The CCU was plugged into the Rover which was a portable desk-sized base station that AT&T brought to trade shows to show off CDMA service.

The Qualcomm simulator was replaced with the AT&T Bell Labs code that was designed to the same Qualcomm CDMA modem API.

We executed the Design Verification Tests (DVT) to confirm that the the Qualcomm modem actually worked as required in the development contract. I was responsible for the integration of the two systems and was driving necessary changes to get the system to work over the course of the two weeks.

After Bell Labs was happy with the new modem and the Qualcomm engineers went home, came the task of integrating the modem and code on the CCU with the code that the rest of the 150 developers had been working on for the base station.

This was called the CDMA Cluster Integration Cluster Integration where all the other subsystems (CCC, RCC, etc) were integrated to produce a fully-functional cellular site and of course the cell sites needed to be integrated with the 5ESS which controlled the cellular system.

The Bell Labs code was fresh, very little of it had ever been compiled and even less had been thoroughly tested. Since I was the only one on the team who had ever worked on the low levels of the software, I was called in again to get the integration going. First I had to create correct makefiles for the system and had to debug and fix the bootstrap code. Next the channels (Pilot, Sync, Paging, Access, Traffic) had to be brought up one at a time. The code had to be layered in so that we could isolate problems. We found both software and hardware problems along the way. As we got further along we integrated with more and more of the system until we actually placed calls with the 5ESS.

FDDI Network Interface Card Design Verification Testing

Prior to my contract with Lucent I worked at Formation, Inc. of Moorestown, New Jersey on a contract through Computer Horizons, Inc.. The first project I executed for them was designing and executing the Design Verification Tests required to satisfy the requirements of their development contract with an aerospace company.

The VME FDDI network adapter card used a Motorola 68030 microprocessor. Formation was responsible for developing the hardware, and the software that allowed the customer's application to transmit and receive on an FDDI ring.

The test arrangement I designed used a Formation stock 68020-based VME CPU card as a controller, a memory card, and multiple FDDI cards. A test kernel that could execute primitive instructions was designed for the VME card. I designed a script interpreting test controller on a PC that communicated with the VME CPU over a serial port.

The tests started at the lowest level an confirmed proper symbol exchange with an FDDI test set. Basic line disciplines were also confirmed. Once these tests were confirmed, a ring with several cards plus the test set was created. The VME CPU would set up data patterns on the memory card and trigger transmit and receives between the cards over the ring. The test set confirmed that we met throughput and other requirements.

The test documentation was a deliverable for the customer per the contract. The text gave a high level description of the purpose of each test. Details of the test execution was contained in the scripts and the comments in the scripts. As the scripts executed the test system displayed the results of each test step on the screen. If a manual user operation was required (like moving a cable), the script could provide instructions and wait for confirmation from the operator. The complete regression test would execute about 3,000 test cases in about 15 minutes.

Redundant Array of Independent Drives (RAID) Storage System

The second project I worked on was design verification of a RAID disk system Formation was designing. This product not only had redundant sets of drives, but it had redundant controllers, redundant host interfaces to the controllers, and redundant power supplies. The product was designed to support any host with a Fast and Wide SCSI interface, but the initial customer was going to use it in an array with RS6000 computers running the AIX variant of UNIX.

The RAID system could contain up to 36 drives on six SCSI strings. A controller was situated at the top and bottom of the strings. If one failed, the other was to take over. To qualify as a RAID device, certain performance gains must be met. For instance, a 5+1 RAID3 array should have read and write performance no less than 5 times faster than the individual drives. Caching algorithms within the system can improve the performance as well. Additionally, the SCSI interface needed to be confirmed as valid and compatible with the AIX drivers.

Since a lot of performance metrics needed to accumulated I needed an automated test system of some sort. I ported the test system from the previous product to an RS6000. In order to guarantee that the software was not the bottleneck, I made the system fork multiple copies of itself which communication through shared memory. I had to bypass the file system and work with the SCSI drivers directly in order to keep the throughput high and get accurate timing information. The RAID system had a serial console port that was used for configuration and monitoring. This was brought into the test control system so that closed loop testing could be performed.

PC-based Manufacturing Test System

Prior to my Formation contract I worked for Integrated Network Corporation. INC made DDS modems and related items which were manufactured in Korea. Whenever they had a new or changed design, they would also design a test fixture to ship to the factory as well. A typical test fixture consisted of and MSDOS PC with parallel I/O cards installed, cabling to a test mount for the unit under test, and the controlling software.

I was assigned to develop a test fixture for a series of new adapter boards that converted TTL logic levels to various interfaces like RS232 and X.21. In all there were to be several adapters each with its own test set. The hardware would have to allow closed-loop testing so that it could be fully automated.

The PC test program was to have a look and feel of an existing test set. After examining the code, This code communicated with different types of parallel I/O cards plugged into the PC bus. The screen was a character-graphics display. There was a textual text menu selection area, a scrollable message area, and a simple character-graphic area with representation of front-panel LED states, DIP switch settings, and other indicators of how the unit under test was supposed to behave.

I determined that the structure was too product-specific and not portable enough to allow efficient support of multiple test sets. The graphic and I/O code was hand sculpted and there were no abstract reusable modules that were useful as is for a new test set. I proceeded to design an abstract test execution engine using object-oriented techniques taking advantage of the common attributes of test steps, screen indicators, and I/O devices.

A test procedure is a series of test steps. A test step involves setting up a set of initial conditions and then waiting for a set of expected outputs. I defined the concept of an I/O bit and an I/O port. An I/O bit was a single physical pin on an I/O card. One or more could be defined as being members of a logical I/O port. The ports were given meaningful names by the test script procedure. The I/O pin was given a generated name. The name was generated by the device driver layer. The test procedure would write values to sets of I/O ports, the device drivers would distribute the bits to I/O pins, and the values would propagate to the unit under test. The test procedure would then compare the read back values of other I/O ports to expected values. These values would be constructed by the device driver layer from the individual I/O pins.

Likewise character-graphic indicators of LEDs, illuminated labels, LED arrays, switches, numeric displays, etc, were treated in a generic fashion. When defined, they would be associated with an I/O port, they would respond to changes in value.

Various canned test algorithms could be applied to I/O ports. For instance you could define an interconnection net list (including logical inversions) in terms of I/O ports. You could then specify that a walking ones, or walking zeroes test be executed to check for shorts. After each output port change, the proper state on the inputs would be verified.

Following this method, a great deal of time in design a test system was saved. Before this system existed, most of the test development time was spent confirming what wires were connected to what cable to what I/O device at what address or fitting the screen displays by hand rather than designing the best algorithm for testing the unit under test. Small changes in the design of the unit under test could cause sweeping, un-manageable changes in the test set design.

With the new design, the test algorithm was expressed in terms of logical I/O ports. These ports would be named in terms of the I/O interfaces of the unit under test. The test developer could easily confirm the algorithm design with the designer of the unit under test without getting bogged down with how the physical connections were actually made.

The construction of the test set was much less critical as well. Since all standard PC interfaces (e.g. parallel printer port, serial port control and status signals) many simple test sets (for instance a Null Modem Adapter tester) could be made without opening up the PC chassis. If additional cards needed to be added to get the required number of I/O pins, they could be plugged in at arbitrary I/O addresses since only the base name passed to the device driver in the configuration file mattered.

A common mistake was to incorrectly wire a wiring harness to the unit under test. For instance the connections to a DIP switch were reversed. In the old method either the wiring would have to be fixed, or the bit-banging code would have to be modified to fix the problem. Both are time-consuming and error-prone. With the new system, only the order in which the I/O pins were assigned to the I/O port needed to be reversed, and the test set schematic needed to be updated to how the wiring was actually done.

Once the system was ready, I had developed about 15 test sets. Only a few were actually for manufacturing operations. The rest were for testing various cable assemblies used in the lab. It was faster and easier to change the configuration files to test a new cable than it was to buzz out the cable with a meter.

Telecommunications Alarm Surveillance System

One contract I had was for AT&T Networks Systems. They had a maintenance contract for a product called the General Telemetry Processor (GTP) that I had worked on years earlier. Their customer, AT&T Long Lines, used the GTP for Alarm Surveillance and Control (AS&C) of telephony equipment and Performance Monitoring and Fault Locating on fiber optic transmissions systems in the long distance network.

The GTP was a component of the Telecommunications Maintenance and Administration System (TMAS) Operations Support System network. GTPs reside in long distance telephone central offices collect the alarm and performance data from transmission equipment using parallel inputs as well as the Telemetry Byte-Oriented Serial (TBOS) and Telemetry Asynchronous Block Serial (TABS) protocols. The GTP did change-of-state processing of the alarm data and reduction and binning of the performance data. The data was reported on demand to the TMAS central using the E-Telemetry communication protocol.

The GTP did not have any local display terminals. It's data was only visible at the transmissions maintenance center. Technicians there would review alarm reports and then arrange for dispatching service personnel to the location with the failure. While the equipment is repaired the service technician would have to talk to the TMAS technician to see if the repair efforts had cleared the failure. Since one TMAS location monitored half of the United States at a time, the TMAS technician's work load was high and they may not communicate effectively with the field technicians. As a result network problems may take much longer to fix than necessary and at a much greater expense than if the field technician could see what is going on in real time.

AT&T requested the Network Systems maintenance organization to add a local alarm display capability to the GTP. I was called in for the development. At first they were only concerned with local display of a few alarms that were critical to network operation. In particular power, fire, and security alarms. We designed out a simple solution of code changes to the GTP which would spit out the desired changes out a serial port to a new display panel I would design. There was still a problem with how to set up a database within the GTP to support the function.

After presenting the idea to the customer, we discovered other problems they were trying to solve at the same time. The GTP monitors about 20,000 alarms and only a few (as requested in the original development contract) could be supported by the proposed change in a reasonable fashion. Also, a typical central office would have an average of five GTPs installed in a rack, so a local technician would have to look at multiple display units to see what was going on.

We decided to advance to a PC-based design. This approach had several advantages. First, the GTP had limited memory and so might not be able to hold software of the sophistication required for this project, especially when considering that the requirements of the feature were growing or changing daily. Moving the operational code from the GTP to an external computer also limited changes to the GTP which prevented the chance of corrupting the operation of a critical AT&T network component. We could also then support an arbitrary number of GTPs and an arbitrary database size simply by right-sizing the PC.

Further requirements came in. It turned out there were a lot of alarms that were not monitored by the GTP, but instead were monitored by older E-Telemetry remotes. Where it would be possible to make some changes to the GTP software to support the local alarm display feature, it would not be possible with the older remotes because the source code or hardware interfaces were not available. The only way to see their data would be to listen in or eavesdrop on their conversations with the TMAS central. To support this, we added to the PC E2A Telemetry cards I had designed years ago for the ETEST E2A Telemetry Test Set. With two cards, I could monitor both sides of the communications between all of the remotes and the central on a single network.

A problem with eavesdropping on the E2A Telemetry link is that when the central stops polling for data, the remote stops reporting. So, if a facility failure caused the link to the TMAS central to fail, the local alarm display system would not be able to access the data required to solve the problem. To solve this problem, I designed an external rack-mount chassis. For each E-Telemetry facility supported (up to four were required) there was a voiceband 4-wire bridge, two 202T modems, and a relay. The modems were plugged into a pair of E2A cards installed in the PC. One card monitored the polls from the central, the other monitored the responses from the remotes. The relay allowed the system to disconnect the central from the local office in the case where the TMAS polls have stopped and the local display system took over. An additional set of relays were used to drive aisle alarm lamps.

By now the system was named the Local Alarm Display System (LADS). We now had the ability to monitor the activity on approximately eighty E-Telemetry remotes for a total of about three hundred thousand alarm points. Now we had to solve the databasing problem. The E2A protocol is a very efficient binary protocol. For each point that was to be monitored by LADS, descriptive information needed to be entered. This would be the most expensive part of an installation if it had to be done manually. Instead, I investigated the TMAS database which contained the master database. It supported about 14 fields for each alarm point it knew about. I proposed an installation procedure where a field technician would dial up with a modem from the remote location where the LADS was installed. A Korn Shell script was executed by the technician and the identities of the GTP or other remotes were entered. The script then executed my SQL scripts to extract the raw database information from the Informix database. A typical report was many megabytes; much too large to download with a 2400bps modem. I wrote several post-processing programs to convert the alarm records to reduce redundant data. The scripts then compressed the result and packaged the files for download. The result was downloaded to the LADS PC with a terminal emulator program, unpacked, and then used.

And of course, one more request was made, remote terminals. Since telephone central offices were typically several stories high, they wanted to be able to have visibility not only where the LADS PC was, but also at various sites throughout a building. And of course, managers wanted to be able to dial in from home to see what's going on in their building. We wanted to keep the remote displays' look and feel as similar to the LADS PC display as possible to simplify training and software reuse. We searched for suitable remote terminals. Most terminals were VT100 variants. These did not have keyboards that matched the PC layout and the character sets were not compatible. Advanced Display Systems (ADS) produced a PC-Terminal that did meet this requirement. It also could be ordered with an RS422 interface which we would need since the terminals were typically going to be placed well beyond the reach of RS232.

The software architecture was the result of a lot of choices. The operating system choices at the time were Unix, MSDOS, Windows 3.1, and DesqView386. In order to support the 20 or so serial interfaces, a real time operating system needed to be chosen. MS-DOS would only be useful to load another operating system that supported our requirements. It's programming architecture is not suitable for such a system. Windows 3.1 had just come out, and was not real time though it did support some multi-tasking. Mult-tasking stopped whenever you accessed menus on the GUI. If someone entered a menu and then went to lunch, they would basically disable the system.

DesqView386 and Unix were the only possible choices. There were no mature PC-based Unix versions at the time. Also, source licenses would have to be negotiated since we would have to write our own drivers in many cases. Such legal work would both blow our financial budget and our time budget.

DesqView386 was chosen. It was a true-multitasking operating system designed specifically for the PC environment. Since they designed it to provide Virtual-MSDOS sessions, no special programming techniques were required for the applications. It also supported hooks for grabbing and filtering the input/output streams of application windows. As an added advantage, it took no special skills to install and had flexible and reasonably cheap licensing, especially when you consider we would have a production run of about 5,000 units to a single, company-internal, customer.

The software architecture also has to support the skills of the development staff. I was the only developer assigned, though the programming load was in excess of what I could produce. I was promised the occasional help of random people in the organization, many of which were junior programmers and only had Unix experience. They had no embedded, real-time, or communication protocol experience. I determined that all I would be able to count on additional help for would be generating alarm reports.

With this in mind and with the full expectation of rapid requirements changes with respect to the data displays, I decided all of the reports would be initially designed using a scripting language, and then converted to C when the final layouts were decided upon and we decided that we needed to increase the speed. To support rapid report development and efficient use of the Unix programmers I decided to use the MKS ToolKit in the shell windows for the system. The MKS ToolKit provided all of the standard Unix tools (AWK, grep, various shells, etc.) on the PC environment. The reports could be generated with any method they were used to using on Unix.

The remote terminals needed some security features. I used the MKS login program to provide security. A user could not access the system without logname and password matches. I created a getty-like process for each remote terminal serial port. This process loaded the MKS login program when it detected that the remote terminal had become active. If the user name and password tests passed, the Korn-Shell was loaded and the user was given system access as restricted by the passwd file.

Upon startup, a series of E-Telemetry Eavesdrop drivers were added to the E-Telemetry interfaces. These drivers compared request to responses and determined which alarms on which remotes were changing state. They used a hash to convert the facility/remote/display/bit telemetry address to a database index. Changes were recorded into large circular queue that cached bursts of events (bursts of 7,000 events were not uncommon when a fiber-optic cable was cut). It took about twelve bytes to store the event identifier and timestamp in the queue.

The alarm queue data structure was a circular list. It had one write location and supported subscriptions for several read pointers. If there was a situation where the write pointer caught up to one or more of the read pointers, the read pointers would be pushed ahead, effectively deleting the oldest events in the queue. Since the event records did not require much storage, many thousands of records could be stored in the queue and so event dropping only occurred during the busiest times. The read pointers were serviced first-come, first served. The highest priority subscriber was the logging task which saved the raw events to the disk. This prevented loss of events

Scrolling event screens subscribed to the event queue. They could be configured to look at all or only a subset of the data, for instance a terminal in a power room might only display power alarms. The scrolling log would extract the alarm identifier from a log record and use it to retrieve an alarm descriptor record from an indexed file. This descriptor contained indices for 14 database fields, each field was stored in a separate file. Depending on the purpose of a given event log screen, different fields would be used.

Since retrieving and displaying the text fields is time-consuming, there is a possibility that an event burst could produce more events than could be displayed before being dropped from the event queue. In this case the screen would have one or more gaps in what was displayed and an indicator would show that display information was lost. This was not considered to be a problem since in cases of event bursts the screen would scroll too fast to be read anyway. Since the logging task kept up with the bursts, the complete event history could be reconstructed from the event logs on disk.

Also upon startup a series of "getty-like" tasks were created. These created DesqView windowless applications. They took advantage of the standard input and output hooks that DesqView has to replace the keyboard and screen I/O with I/O to one or more serial ports. Each getty would wait for the Data Set Ready (DSR) and Carrier Detect (CD) leads to become active indicating that there was an incoming modem call, or a terminal behind a null modem had been turned on. The getty then loaded the MKS login program. The login program functions like the UNIX login program. It prompts the user for a user name and password and checks /etc/passwd for a match. Typically a match would cause login to for the Korn shell for the user. The /etc/passwd file was used to segregate users of different types. For trusted users the path was set up so that most tools were accessible and all of the remote versions of the logs and reports were available. Lower leve users were given a restricted shell which prevented them from changing their working directory and their path was restricted to prevent them from being able to execute programs that were not appropriate for them. The passwd file could also be set up to not provide a shell at all, but just to run a pre-defined report. This added security was useful for some dial-up applications and stations where only display was required.

The goal was to sell and install as many as units as quickly as possible, so the manufacturing and delivery processes were streamlined to take into account the resources we had available from AT&T Network Systems for the job. For time reasons, it did not make sense to have the AT&T factories manufacture the backplane for the modems and bridges, instead, I worked with Dantel and had them design a backplane and frame arrangement for us that could be ordered on demand. They could populate the slots with the cards or we would order and install them ourselves as needs varied for different sites. Dantel would drop ship units directly to the field locations for installation.

The PCs were configured in the AT&T Network Systems plant in Columbus, Ohio. We used off-the-shelf 80386-based AT&T computers. The computers arrived with MS-DOS installed and the factory workers added DesqView386, MKS Toolkit, and our application code. As needed, they would also plug in the E-Telemetry interface cards and expanded serial cards. They would then package it back up with the necessary cables and ship it to the customer site.

The customer would locate a rack near the telemetry facilities for the modem rack. The telemetry facilities would be routed through the bridges. The computer would be located at a nearby table. A phone line would be installed so that the operator could then dial into the TMAS system and execute the database extraction programs for that site. The files would be installed, the application would be started, and everyone would go home. A complete installation would take about a day.

Trans-Atlantic Cable Surveillance System

About the time that I developed LADS, the first fiber-optic trans-Atlantic cable was to be installed. This cable was called TAT-9 and was installed by a consortium led by AT&T. The cable left North America from New Jersey and Newfoundland. These two legs were spliced under the ocean and the joined cable continued to a point south of England. At this point the cable split and legs went to England, France, and Spain.

In the past, maintenance of the cables was not sophisticated. Primarily only binary status and alarm indicators were available at the cable heads and these were easily integrated with whatever alarm system was present in the office. The digital cables being installed were based upon existing AT&T FT-Series-G transmission systems. These systems were designed to transmit their alarms as well as digital performance monitoring information through GTPs to TMAS. This was fine for AT&T at it's cable head in New Jersey, but was not useful for the foreign partners. The cable contract required that this data was to be available to all partners. The initial assumption was to give the partners terminals to the AT&T TMAS system. This was rejected by the partners. The only second possibility was to set up TMAS systems in each of the foreign countries and only have them monitor using one GTP instead of the dozens it normally would. This was too expensive. They needed a cheaper and more universal solution.

The group I worked with worked closely with the fiber-optic transmission developers. They found out that LADS was being developed to provide local alarm display features through the GTP and wanted to know if it could also provide the performance monitoring data required for the cable project. Oh, and by the way, it was needed in two months.

Well, the GTP collects the alarm and performance monitoring data from the transmission terminals using the Telemetry Asynchronous Block Serial (TABS) protocol. In particular, TABS-AS&C and TABS-PM. It also provides a Fault Locating application with TABS-FL, though that was not needed by TAT-9. LADS was designed to collect only alarm information, and only from the E-Telemetry interface. It did however have a flexible software and hardware architecture that was a good basis for a new design.

The Performance Monitoring Alarm System (PMAS) was born by taking a standard LADS system, stripping away the E-Telemetry protocol handlers and the external chassis that interfaced with the E-Telemetry facilities. The E-Telemetry interface cards were replaced with versions of the multiple serial port cards that we used on LADS for remote terminals that had the RS422 interface that the TABS interfaces required.

Behind the RS422 serial ports TABS-AS&C and TABS-PM protocol stacks were created. The AS&C mapped well into the event structure that the E-Telemetry protocol stack used. The PM data had to be encapsulated into an new event record type for the event queue. The alarm events could be displayed using the same scrolling and history display code used for LADS. The main difference was that a database generation tool had to be created to compensate for the fact that we could not download if from TMAS. New Performance Monitoring scrolling event and history displays were created for the PM data.

Since there were only five sites for this product, we manufactured the PCs in our development labs. The software installation was similar to what I described for LADS. In fact, we prototyped the LADS factory manufacturing procedures with PMAS since PMAS was given a higher priority and so was deployed ahead of LADS. The PMAS systems used a different set of plug-in cards. Also, we could not use identical PCs since some required 110 volts and some required 220 volts. The dial up interfaces were a pain since the five countries have vastly different rules for modems. This required slightly different software for the gettys in different countries.

Modem Test Unit

Prior to becoming a consultant I first worked for AT&T Bell Laboratories and then for Telecom Analysis Systems (TAS). TAS developed telephone channel simulators that produced realistic impairments that you would expect to encounter on telephone facilities. You could configure them for different mixes and amplitudes of impairments. These channel simulators were perfect for manufactures of modems (such as Hayes).

I worked at Telecom Analysis Systems as the lead engineer in the development of the Gemini Dual-Terminal Emulator modem test unit. This was basically a dual-channel BERT which allowed you to control an analyze the data traffic through a pair of modems to determine the modem's performance over varied telephone line conditions.

I designed the electronics, and ran the three-person team that developed the Xinu operating system based application software. In addition I participated in the formation of several other TAS products and coached the other teams to attempt to achieve software portability, reliability, and re-usability.

AT&T E-Telemetry Alarm Surveillance and Control Systems

My original career was as a direct hire to AT&T Bell Labs working for Jay Bowers. I worked on a number of projects such as the telemetry products that provided monitor and control interfaces to the FT3C fiber optic transmission system for the FMAS Operations Support System.