Asynchronous Enterprise Messaging

Here are my notes on messaging applications offered by Microsoft, IBM, and other vendors. This is related to my pages on enterprise development
Let me know what you find helpful or missing.

Topics this page:

Site Map  
About this site  

Asynchronous Message Queuing Services

Unlike electronic mail (e-mail) the messaging discussed here uses different message formats and communication protocols (handshaking mechanisms) to exchange information between software components rather than between people (using software application clients).

Like email, also called "interpersonal messaging", enterprise message queuing applications use asynchronous processing to send and receive messages. An application can send messages when the recipient is not up and running simultaneously. This allows a component to send information to another and to continue to operate without receiving an immediate response.

Email provides a "best effort" delivery method that does NOT automatically confirm receipt.

Transactional Messaging

But transactional messaging systems also assumes the responsibility to ensure reliable delivery of messages to the target queue, or, if it cannot do so, take appropriate action.

Message queuing applications are generally designed so that messages flow in a request/reply fashion. But unlike emails, a transactional messaging client does not have to request messages in order to receive them. Inter-program communication messages are delievered to a client as they arrive.

After the messaging system accepts a message from the application into a queue as an intermediate storage point. The application is then free to continue other work. This allows communicating programs to run at different times.

Message queuing provides the connectivity layer "backbone" for what IBM calls an enterprise service bus (ESB) within a service-orientated architecture (SOA).

Messaging components read and operate on the messages independently. The sending and receiving applications are uncoupled or loosely coupled. rather than tightly coupled, such as with Remote Procedure Call (RPC). The component sending messages can send messages without depending on information about other components' interfaces. Programs are insulated from network complexities (fixed message formats, communications protocols, etc.). Communication protocols are hidden from the applications. All this allows components to be easily replaced.

Most importantly, messaging applications are designed to be reliable — ensure that a message is delivered once and only once. Lower levels of reliability are available for applications that can afford to miss messages or to receive duplicate messages.

Passing SOAP messages through message queues provide visibility and auditability to interactions.

Historical note:

Business Quality Messaging specifications ratified September 1997 by the now defunct non-profit BQM Forum formed April 1997 among AT&T, Compaq, IBM, Intel, Microsoft. aimed to "revolutionize" business-to-business trading by offering an inexpensive and universal method of reliably transferring information, such as SAP R/3 data, between trading partners. WARNING: To avoid viruses, do not visit its website bqm.org or website www.moma-inc.org of the Message Oriented Middleware Association.

Momentum

Talarian

Candle's Roma product

Java Messaging Service API (JMS)

The de facto interface is MQI (Message Queuing Interface) proprietary to IBM. The AMI Application Messaging Interface

Sun developed the Java System Message Queue Service (JMS) application-level messaging standard API SDK to enable J2EE peer-to-peer apps (using Java 2 SDK, Enterprise Edition, version 1.3) to create, send, receive, and read loosely coupled, reliable, and asynchronous messages.

The sample programs for the API 1.1 doc simplifies and improves programmer productivity by defining a common set of concepts and programming strategies that will be supported by all JMS technology-compliant messaging systems.

Sun's free "Platform Edition" has a Windows variant (now) but does not have the scalability, reliability and advanced security features of the for-fee enterprise edition.

JMS operates in two modes when calling a servlet at a JMS Destination:

• JMS Point-to-Point (Queue) and
• JMS Publish/Subscribe (Topic) (supported in MQ with SupportPac MA0C)

JMS messages can participate in Java Transaction API (JTA) transactions.

The workflow:

1. The servlet enqueues messages to a JMS Destination.
2. The MDB (Message Database) receives the messages, and (depending on Message Property settings) will either
   • ignore the message,
   • rollback the message, or
   • generate and enqueue a response message.
3. The servlet receives response messages, verify their correctness, and exit.

Sun's JMS Developer forum

Sun's JMS Tutorial for J2EE SDK 1.3.1 or 1.4.

IBM WebSphere MQ Series (WMQ)

IBM's Message Queue product (developed in IBM's Hurley Labs in the UK) has the largest market share of any messaging middleware solution. Its full name was "IBM WebSphere MQSeries" because it runs on WebSphere servers. Recently, IBM dropped the "Series" for v6.0.

The MQ product consist of three major components operating independently of each other:

1. MQ Server
2. MQ Queue Manager
3. MQS Clients, which maintain synchronous communication / connection with the Queue Manager. It can participate in local but not global units of work involving MQSeries resources.

Servers transmit messages communicate via channels — each a unidirectional link between a MCA (Message Channel Agent) on each queue manager at each end of the communicating channel.

MQ messages can be as large as 100MB with v6.0, its latest release.

Several Queue Managers can be clustered so that each queue manager can access the queues of all other queue managers through Cluster-sender and Cluster-receiver channels rather than remote queue definitions. Information about clusters are held in a SYSTEM CLUSTER REPOSITORY QUEUE and exchanged through a SYSTEM CLUSTER COMMAND QUEUE.

MQSeries uses the Message Channel Protocol (MCP) to ensure delivery.

MQI Calls

Applications using the WMQ API (MQI) issue 13 different calls:
1. MQCONN connects an application program to a queue manager.
2. MQCONNX returns a unique connection handle.
3. MQOPEN opens the object before an inquiry, set, get, or put command could be executed on it

4. MQPMO MQPUT to put messages on queues
5. MQGMO MQGET to take them from queues.
6. MQPUT1 places messages onto a queue that is already open.

7. MQCLOSE closes application objects.
8. MQDISC disconnects connections between an application and a queue manager.

9. MQINQ inquires on the MQSeries object settings.
10. MQSET sets values for attributes of these objects.
11. - 13. Synpoint commands.

MQ Message Descriptors

MQ messages also contain a Message Descriptor (MQMD) which contains metadata about (properties of) the message and its application data:
• persistence,
• priority,
• message id,
• correlation id, which provides a reference to a particular topic
• report,
• feedback,
• reply to queue,
• reply to qmanager,
• expiry,
• group id,
• message sequence number,
• offset,
• format,
• encoding and
• coded character set id

Queue Management Commands

A queue manager can be created using the command:
crtmqm -q QMgrName
(-q specifies the default queue manager)
-lc (use circular logging),
-ll (use linear logging),
-lf (specify size of log file)
-lp (specify directory to hold log files) can be passed with the crtmqm command

A queue manager can be deleted using the command:

dltmqm QMgrName

A queue manager can be started using the command:

strmqm QMgrName
If the queue manager is started for the first time, default objects are usually created.

A queue manager's objects can be inquired about, defined, and altered using the command:

mqsc QMgrName
this accepts standard input and standard output options

Attributes of a queue can be displayed by specifying

DISPLY QUEUE (XXX) ALL

Attribute values can be altered using the ALTER command.

Queues can be deleted using the DELETE command.

Types of Queues

MQ products manage queues (resources):
1. Local queues are local to a queue manager. Thid creates a local queue with a put and get enabled option, a persistence option and maximum depth and maximum message length specified as 1000 and 2000 respectively
   DEFINE QLOCAL (MY_QUEUE) REPLACE + DESCR (" A Local Queue") + PUT (ENABLED) GET (ENABLED) + DEFPSIST (YES) SHARE + MAXDEPTH (1000) MAXMSGL (2000)

2. Remote (Distributed) Queues
   DEFINE QREMOTE (RQ1) + RNAME (YYY) RQMNAME (QM2)
   RNAME refers to the remote queue manager named RQ1
   RQMNANE is the name of the remote queue

3. Alias Queue which references another local queue or a local definition of a remote queue can be defined using this:
   DEFINE QALIAS (XXX) TARGQ (YYY)

4. Model queues are used as a template to create dynamic queues:
   DEFINE QMODEL (ANSQ) DEFTYPE (TEMPDYN)

5. Dynamic queues can be permanent or temporary in that they do not survive a queue manager restart.
   DEFINE QMODEL (ANSQ) DEFTYPE (TEMPDYN)

6. Transmission queue
   DEFINE QLOCAL (QM1) USAGE (XMITQ)

7. A Dead Letter queue is used when either a destination queue is not available or when application data conversion fails at the sending end. There is a SYSTEM DEFAULT DEAD LETTER QUEUE.

Other topics of functionality:

1. Triggering
2. MQGMO segmentation options
3. Message data translation
4. Reporting
5. Functional Sequence
6. Message groups and Segmented Messages
7. Distribution Lists
8. MQ Security

Tek-Tips forum on MQ Series

Training classes on MQ Series from:

• Web Age Solutions
• Verhouf

MQ Series Workflow

Distributed Computing with IBM(r) MQSeries (John Wiley & Sons; Bk&CD Rom edition, October 25, 1996) by Leonard Gilman, Richard Schreiber, Len Gilman

Achieving Production Quality Messaging article by Nancy Cox at Network Computing magazine

PC Networking slideshow

IBM's MQ newsgroup and HTML Forum

Message Queue User Group UK

Other members of IBM's MQSeries family of product provide specialized functionality:

• MQSeries Adapter provides the framework and tools to build and customize MQSeries adapters for existing and new, pre-packaged or custom-developed applications.

• MQSeries Integrator — IBM WebSphere Business Integration Message Broker and WebSphere Business Integration Event Broker (MQSI) — is an information broker providing for message transformation and information flow modeling across multiple and different business systems.

• MQSeries Everyplace interfaces to mobile protocols and devices.

• MQSeries Workflow enables the definition, execution and swift change of complete business processes.

IBM made MQ open for others to integrate with it (the 'glue' that bonds together many disparate platforms and technologies):

• BEA MessageQ
• MDI interface for the Python programming language

IBM's MQWare is for independent software vendors (ISVs) and Internet Service Providers (ISPs) to embed IBM's MQSeries messaging technology in shrink-wrapped Windows NT-based applications. According to a July 1997 news release: The product provides assured, "once only" message delivery between instances of a message-enabled application (e.g., it ensures information is not lost between servers using MQWare to issue "send and forget" messages between instances of their application in an NT Server-based network).

MQWare was the first product to conform to the Business Quality Messaging concept, a set of standard APIs for suppliers to guarantee messaging delivery.

Shanky Tiwari January 2002 article at Developer 2.0

Extended Architecture (XA) Sync Points

Message queuing provide a means of integrating business processes through message-driven processing.

Asynchronous messages owned by various resource managers (managing queues in WMQ or in database tables) can be combined into a single unit of work using the XA transaction management interface published by the X/Open Company.

XA defines a syncpoint coordinator computer subsystem that manages units of work. It determines when updates within a unit of work can be committed or backed out.

IBM's MQI protocol

• MQCMIT commits a unit of work after the program has satisfactorily completed processing. This cannot be used with external coordinators like CICS or IMS.
• MQBACK rolls back an operation to a state before the unit of work began.
• MQBEGIN begins a global unit of work.

Transaction Processing Facility (TPF)

Microsoft's MSMQ Message-Oriented Middleware (MOM)

The Microsoft Host Integration Server 2000 and 2004 MQI Channels Tab is used to define MQI channels.

MSMQ was code-named "Falcon" during its development within Microsoft.

MSMQ offers message queuing functionality such as message delivery, cost-based message routing, and full support for transactions.

Components of Microsoft's Message-Oriented Middleware (MOM) offering include:

• The Primary Enterprise Controller (PEC) for the entire enterprise (with names up to 126 characters long).
• Intersite and intrasite message traffic among sites (with names up to 31 characters long).
• Connecting networks linking independent clients (laptops) with the MSMQ server and the MSMQ dependent client (warehouses/stores of data).

MSMQ is integrated with other Windows NT features such as Microsoft Transaction Server (MTS).

MSMQ interoperes with IBM's MQSeries through the Microsoft MSMQ-MQSeries Bridge and other message queuing products through products from Level 8 Systems.

Triggers

Now-defunct CommerceQuest, Inc. 813.639-6300 (which turned into Proforma, then Metastorm) used to sell MQTester add-in (first available Sep. 2003) which extends LoadRunner scripting, playback, monitoring, reporting and analysis capabilities to the IBM MQSeries infrastructure.

It is "agentless". It issues MQ commands to define triggers which notify events such as Channel Started and Channel Stopped, plus Queue Manager events such as

• Queue Manager Active
• Queue Manager Not Active Remote
• Queue Name Error
• Unknown Object Name
• Not Authorized (type 1 -4)

It also monitors performance triggered events:

• Queue Depth High Threshold exceeded
• Queue Depth Low Threshold exceeded
• Queue Depth Full
• Queue Service Interval High Threshold exceeded

Performance

WMQ vs MSMQ

Microsoft commissioned a study by NSTL [dated May 17, 2000] which claims its Message Queuing (MSMQ) product runs "10 times faster than IBM's MQ Series".