From the Trenches: IBM WebSphere WMB, WTX, MQ
At the moment, I am resurfacing after having spent nearly the last 6 months in the trenches of IBM WebSphere Message Broker (WMB) 6.1.0.9, IBM WebSphere Transformation Extender (WTX) 8.2.0.4, and IBM WebSphere MQ 6.0, a set of products that were inherent constraints in the technology migration project I led at one of my recent clients.
For those who are familiar with IBM WebSphere Application Server (WAS), the flagship in the WebSphere product line, but not familiar with these products, it might be surprising to hear that WAS is just one of hundreds of products for which IBM has chosen to brand "WebSphere".
Many of these products have pre-IBM histories. For example, the first incarnation of WTX was Ascential DataStage TX, and later became DataStage TX and Mercator Integration Broker. Some of this information is readily available, but more than likely one will come across these facts when searching the forums.
Another example of past lives is WMB, which was initially developed by NEON (New Era of Networks), later acquired by Sybase, and initially known later as IBM WebSphere MQ Integrator. Although one will likely come across this information in the forums as well, some processes have retained their "integrator" names, much akin to Orion processes still retained within Oracle Application Server long after the Oracle acquisition, a fact of which I became keenly aware after running across the first exceptions on a past project of mine.
In some very limited respects, I liken the WebSphere product line to the Apache Software Foundation. On another recent project, an individual who had been with my client for quite some time had initially been inquiring about Apache configuration, to which I responded, "Umm, Apache what?" "Oh, Apache HTTP Server". It is very common for professionals to simply refer to "WebSphere" rather than "WAS".
Plenty of resources regarding the purpose of WMB, WTX, and MQ exist out there, so the information you will find here on purpose is brief. However, what I will share is an honest review of what I consider to be some pros and cons of this software suite, based on my experience from the trenches of a recent project.
One aspect that I consider a bit lacking with reviews out there is that those writing the reviews have not actually used the software in any meaningful way. The same goes for the many presentations I have read, which have come in handy to some degree to increase awareness of available product options, but separating the wheat from the chaff can be done rather quickly after some initial exposure is gained.
From my experience for this specific project, when comparing against what can be seen as more manual options, I tend to think that there are benefits to this particular tool suite. To compare these tools to other commercial products, however, is a bit out of scope for this blog post.
Integration with queues and web services is relatively less complex than would otherwise be the case. In addition, I personally think it aids in the development process when processes can be seen visually via the Eclipse-based integrated development environment (IDE).
Speaking of separating the wheat from the chaff, I am cognizant of the fact that using this tooling on what might be considered greenfield development is likely to be quite different than a technology migration, which inherently has significantly more constraints. And quite honestly, there are a host of project factors not related to the actual development of which I need to remind myself are out of scope for this blog post.
Bugs Abound
However, to cut to the chase, let's face it, there are some signficant bugs with this tool suite. Working with these bugs brings to mind the conversations I had a couple years ago with a former colleague of mine who was a Microsoft .NET expert. While I may have led that enterprise architecture project, I was his Java technologies counterpart, and I learned a great deal from him with regard to Microsoft technologies.
One of the many take-aways from my conversations with him was that although Microsoft .NET development staff might be seen as less expensive at a surface level than Java development staff, it is common that Microsoft .NET developers tend to rely on drag-and-drop features of the IDE, and once more knowledge below this surface level is needed due to exceptions or more advanced features, this facade tends to give way to the fact that deeper level expertise is needed.
Handling WMB and WTX bugs was like running a steeplechase race. While it is true I ran a sub-5-minute mile when I was younger, I never ran hurdles due to the injuries my teammates encountered (some of which led to leaving the team), and it pains me to think of what "gifts" the steeplechase could have granted (I still consider running a true gift). Like a steeplechase, however, depite the obstacles and injuries we continued to move forward throughout the duration of the project.
Workspace Corruption
The first bug area was frequent WMB workspace corruption. I have never experienced workspace corruption in all my years using Eclipse. Eclipse users pointed in the right direction as to how to recover from such corruption, but due to the nature of the project, doing as suggested did not really save any time compared to simply creating a new workspace and checking out from source control (in this case, CVS).
The problem with creating a new workspace each time this happened (the frequency of occurence varied considerably, from once every few weeks to multiple times in a single day) is that the project structure was less than optimal, and because this was not greenfield, I was constrained to using the structure despite alternate recommendations to simplify maintainability.
Of course, recognizing when the workspace became corrupted was an art in itself, because it came in many forms. With WMB, one can build deployable broker archive (BAR) files using the IDE or the console via the command "mqsicreatebar". Building via the IDE provides more feedback, but unfortunately it also provides very little flexibility in terms of configuration, so what we ended up doing for each work product was move to the command line once core pieces were successfully built via the IDE.
Command "mqsicreatebar" provides significant benefits over the IDE, but builds are either successful or end in with an error code of 99, with no explanation. The manual that IBM provides offers no help, other than to state the obvious. Along with these error codes, there are also what I came to call sub-codes, so it appears that the compiler attempts to offer some additional feedback, albeit with no reference to interpret.
Workspace corruption tended to end with a sub-code of 13. The only way we figured out how to resolve this error was to restart WMB Toolkit (the name of the IDE). Simply performing project cleans had no positive effect. Early on, before I realized how little feedback I would get upon error, some single-digit sub-codes were also received, but these were most often due to typos in the override properties files that can be provided in conjunction with "mqsicreatebar". The exception was a sub-code of 2, which evidently signaled a successful build, with subsequent successful unit tests, but the tooling complained nonetheless.
Odd Behavior
The second bug area is associated with the odd behavior of WMB Toolkit and WTX Design Studio. Examples abound, but two immediately come to mind. What is interesting about at least a portion of these bugs is that contributors to the developer forums dismiss these as being "well known" by the community, with some that persist over several releases of the product, but never seem to get fixed. Developers using this tool for the first time will long for open source software with strong communities that have a mission to provide continuous improvement.
One example is the annoying behavior of WTX Map nodes in WMB Toolkit. Each WTX Map node points to a compiled map created by WTX Design Studio. If the compiled map represented by the underlying WTX Map node property does not exist, the output "terminal" of the node will disappear, and the outgoing flow originally associated with the previously existing terminal will be flagged. Even worse, the WTX Design Studio perspective of WMB Toolkit and WTX Design Studio itself decides seemingly sporadically when it wants to clean projects, often at inconvenient times.
The best example of this behavior is when a BAR has just been built and deployed in the local development environment. WMB Toolkit provides a convenient initial local unit test environment, but when unit testing commences after this first deployment, execution will fail. At first experience, the reason why this execution first fails is a complete mystery. I remember reviewing all of my work, step-by-step, the first time this happened to me, to determine what had gone awry.
Because of this first experience, and many, many other similar situations since, I have become an expert at reading trace logs, which came in handy because client developers were accustomed to using the graphical debugger alone, a convenient but incomplete debugging solution. Of course, the trace log facilities typically consumed 50 GB to 100 GB on a daily basis, so removing trace files each morning became a daily habit.
After deployment, WTX maps to which each of the associated nodes are pointed need to be recompiled. After this recompilation, the nodes are then able to find the binaries for these maps again, but even though output terminals of nodes reappear, the outgoing flows need to be manually dragged back to these terminals. A developer should not need to do this on so frequent a basis, and when multiple WTX map nodes exist within a given WMB flow, this process is exceedingly ardous, and frankly, a waste of time that would never survive a Lean development process.
Another example of odd behavior appears to be some type of workspace caching issue, and this problem arises in several different forms, with two being the most prevalent. The first is associated with type trees created in WTX Design Studio. In order to map field values from one file definition to another, whether each is an XSD, flat file, COBOL copybook, or one of many other possibilities (one of the strengths of this tooling), a type tree with an MTT file extension needs to be created for each file definition.
However, after the initial creation of an MTT, the type tree can be further tweaked at the properties level in a host of different ways. The problem is that in many scenarios, the modified type tree may not be immediately available in the workspace even after WTX Design Studio signals a successful build. As a new user of this tooling, I was initially inclinded to think that I had done something wrong when availability of a new type tree did not seem to exist. After a lot of trial-and-error, it was determined that a project cleaning, in addition to restarting the IDE, was needed in order to fix this issue.
After a while, we became accustomed to this bug, but a related, more serious bug was when we occasionally saw type trees from other unrelated projects become available from a given project. The fix was the same. Project cleaning is not enough. Restarting WTX Design Studio was also necessary.
Poor Documentation
I extend the definition of "bug" beginning with the third bug area. At first glance, or even after some light usage following the building of initial proof of concepts, documentation for this tool suite appears to be thorough. And in fact, I consider some of the documentation rather good. For example, the "Functions and Expressions" manual for WTX is reliable and provides some good examples for creating data rules.
However, the reliability of some isolated manuals does not rub off on other manuals and documents. Before discussing further, be aware of the fact that when I discuss poor documentation here, I exclude qualms that are related to documents that are somewhat deceptive due to not being available for a specific release of a product, or those that do not at least flag readers to the possibility that a document as currently written may not provide screen shots immediately pertaining to the stated release reference because of enhancements that may have been released along with some mid-release bug fixes.
For example IBM Redbook "WebSphere Message Broker Basics", the book that I first read to ramp up on WMB Toolkit, includes quite a few misrepresentations, even though at first glance it appears to be a complete, step-by-step introductory guide. I still ended up creating some initial proof of concepts based on this particular guide, but it was arduous at times to figure out the actual way to do things in WMB Toolkit that were explained incorrectly.
Examples of how the actual IDEs are not self-documenting also abound. If it were not for the urgency of the enterprise that my team complete the technology migration within the specified time period, I can easily imagine a typical developer not coming across many of these issues until after years of usage, after the tooling has already engrained itself in the organization.
One example is associated with the type tree functionality mentioned earlier in this blog post. Specific to COBOL copybooks, what I consider a prominent bug is related to 88-Levels. For those not familiar with 88-Levels, COBOL provides a mechanism within a copybook file definition that essentially permits a program to refer to the specific value of a variable by using a specified name. Essentially an alias for the value, this mechanism is intended to help prevent hardcoding of commonly used values within the code.
However, WTX Design Studio in my mind has corrupted the purpose of this language mechanism. After unit testing a transaction that contained an 88-Level definition in one of the associated file definitions, I became quickly aware that 88-Levels actually restrict the corresponding data values for the associated variables. When browsing properties for such a type tree, you will notice the "Restrictions" property, which has a nested property called "Rule" that can be one of two values, "Include" or "Exclude".
Since an 88-Level is not intended to be a rule, per se, when confronting a map that fails because of data provided for an 88-Level field, one might be inclined to choose "Exclude". However, this option does not exclude the variable as a rule. What this option does, in the words of the "Type Designer" manual, is "exclude the restriction values as invalid data" when "Exclude" is chosen, and "include the restriction values as valid data" when "Include" is chosen.
When explaining this issue to the client developers, they did not understand the issue. The problem is that 88-Level fields are intended to simply provide convenience and maintainability, and are not intended in any way to somehow restrict the values that the field can hold. The only options that WTX Design Studio offers in this regard are to include or exclude restriction values, but the 88-Level values are not intended to be restriction values.
The only option is to re-create the type tree after removing 88-Level entries from the associated COBOL copybook. However, this brings up another issue. Should it really be considered a best practice to start manipulating legacy file definitions? With greenfield development, removal of 88-Level entries might be considered a valid option, but not if traceability to source systems is desired. This issue remains logged in the JIRA issues database I set up for the project, with no clear path to resolution.
The lack of effective documentation for command line commands such as "mqsicreatebar" was already mentioned earlier in this blog post. While not a rampant issue, there are quite a few cases in the tooling where the developer can be misled down the wrong path until unit testing is actually performed to determine tooling behavior. The best example of this that comes to mind is associated with XML validation.
After creating a map within WTX Design Studio, an option called "map settings" can be accessed either in WTX Design Studio itself, or via a WTX Map node within WMB Toolkit that points to the map. One aspect of map settings to which we continually returned was the level of map tracing that was needed. Map tracing is valuable because it provides detail behind map errors, similar to the flow-level tracing that was mentioned earlier in this blog post.
Map settings also provide other properties that can be customized based on need, and one setting that we eventually used, before incorporating XSLT nodes, permitted the ability to turn off XML validation. However, discovery of this option was not trivial, and was so inappropriately named that I am not able to find this setting as I write this blog post. The apparent ability to turn validation on and off at the SOAP Request node level is also a bit esoteric.
Small Community
Last of all, the developer community surrounding this tool suite can be considered another bug area. As mentioned earlier in this blog post, "developers using this tool for the first time will long for open source software with strong communities that have a mission to provide continuous improvement". In addition, the forums associated with this tool suite seem to be dominated by a handful of unsupporting individuals that routinely request that those seeking assistance refer to the IBM product information centers or Redbooks before attempting to request for help. A commercial product with paying customers should simply be better equipped to help. And because much of the assistance sought concerns matterns that should be easily ascertained from the tooling, it is likely that this tool suite exhibits an unnecessarily high level of complexity.