IDevResource.com COM channel - COM Pipes by Richard Grimes

The Developer's Resource & Community Site

COM	XML	ASP	Java & Misc.	NEW: VS.NET
International	This Week	Forums	Author Central	Find a Job

COM Pipes

Author: Richard Grimes
Date Submitted: March 22 2000

What are COM Pipes?

DCE RPC has a concept called pipes; the idea is that a client and server set up a data transfer conduit between them for efficient transfer of data. RPC pipes are far from straight forward to set up because they involve defining the pipe and the data that will be transferred, as well as memory allocation and deallocation routines and, of course, the code to do the actual transfer. However, it is important to point out that RPC pipes are defined as an RPC mechanism and they are not the same as the networking concept of named pipes (although since RPC can be layered over named pipes, they may use named pipes as a transport).

COM pipes are available in Windows 2000 and are the COM equivalent of DCE pipes. They are used to transfer large buffers of BYTEs, longs or doubles, and to this end COM defines three interfaces: IPipeByte, IPipeLong and IPipeDouble. COM provides the marshalling code for these interfaces in OLE32.DLL. On the surface they look quite innocuous, here is IPipeByte:


[  object, 
   uuid(DB2F3ACA-2F86-11d1-8E04-00C04FB9989A), 
   async_uuid(DB2F3ACB-2F86-11d1-8E04-00C04FB9989A), 
   pointer_default(unique)]
interface IPipeByte : IUnknown
{
   HRESULT Pull(
      [out, size_is(cRequest), 
         length_is(*pcReturned)] BYTE* buf, 
      [in]  ULONG  cRequest, [out] ULONG* pcReturned);
   HRESULT Push([in, size_is(cSent)] BYTE* buf, 
      [in] ULONG cSent);
}

The first thing to point out is that pipes can be called synchronously, or asynchronously. The next thing to notice is that pipes are bi-directional, so that once created they can be use to pass data from the pipe implementation to the data consumer, or from the data consumer to the implementer.

However, there is more to pipes than just this. The pipe marshalling code provided by Windows 2000 implements read-ahead. What this means is that when a pipe user has received a buffer from the pipe implementer COM will call the implementer again for another buffer's worth of data before the caller makes this request.

Since large amounts of data will be pulled it can be assumed that once the data consumer has got one buffer's worth of data it will have some finite processing time before it is ready to ask for more. However, the next data buffer will take time to be transmitted over the network so it means that there is a double performance hit for the application. The first hit occurs when the consumer is processing the data because the data source will effectively be idle, because no call is active. The second hit occurs when the consumer makes a subsequent call to get more data, at which point the consumer thread is blocked while the data is transmitted.

Since COM pipe marshalling uses read-ahead, it means that while the consumer is processing data from a previous call, COM is already making a call to the data source, hopefully so that when the data consumer makes another call COM will already have received the data ready for the consumer to process. Of course, you could implement some mechanism like this yourself with asynchronous COM (also know as non-blocking calls), but it is not straightforward.

Note that COM just provides the marshalling code, it does not provide any implementation for the actual pipes. The developer has to write this code, both for pulling and pushing data. There are two ways to use pipes, depending on where the pipe implementation resides: in the client or in the server. If the client implements the pipe it needs to have an object that implements one of the pipe interfaces and then it should pass this to the server component as an [in] parameter on an interface method call.

Figure 1: Client implements the pipe object, the server will call methods on the client's pipe.

On the other hand the server can implement the pipe, in which case the pipe interface is returned to the client via an [out] pointer and so the client makes the calls to the pipe.

Figure 2: Component implements the pipe object, the client makes calls on the server's pipe

To a large extent whether the client or server implements the pipe is moot because as I have already pointed out, a pipe implementation can be used both to push and pull data.

To aid read-ahead pipe implementations must indicate when there is no more data to transfer. To do this an implementation of Pull() must return the data requested in cRequest until there is no more to be returned, and return the actual number of data items transferred in the pcReturned parameter. COM will continue to call Pull() during read-ahead until it gets a value of 0 via pcReturned. This means that there will always be one extra call to the pipe implementation: COM does not assume that because *pcReturned != cRequest there is no more data. This is a good thing because a data source could be generating data and may not be able to do this as fast as COM is making the calls to Pull().

An implementation of Pull() could look like this:


HRESULT CMyObj::Pull(BYTE* pBuf, ULONG cRequest, 
                     ULONG* pcReturned)
{
   if (bNoMoreData)
   {
     *pcReturned = 0;
     return S_OK;
   }
   *pcReturned = GetDataFromSomewhere(pBuf, cRequest);
   return S_OK;
}

where GetDataFromSomewhere() is some method that generates the data. It could be called like this:


ULONG ret;
do
{
   // ask for 1000 bytes to be sent at a time
   pPipe->Pull(buf, 1000, &ret);
   if (ret) ProcessData(buf, ret);
}
while (ret !=0)

where ProcessData() is some method to do something with the transferred data.

The Push() method is used to send data from a user of a pipe to the pipe implementer. Again, COM will manage the network calls for you, buffering the data so that the pipe caller can continue to send data even if the pipe implementation is still processing a previous buffer. An implementation of Push() could look like this:


HRESULT CMyObj::Push(BYTE* pBuf, ULONG cSent)
{
   ProcessData(pBuf, cSent);
   return S_OK;
}

and it could be called like this:


// send first 1000 bytes
pPipe->Push(buf, 1000);
// send remaining 500 bytes 
pPipe->Push(buf + 1000, 500);
// finish transfer
pPipe->Push(0, 0);

Here, the caller is passing 1500 bytes in batches no larger than 1000 bytes. The final call to Push() is required to inform COM that no more data will be transferred.

Since pipes are bi-directional, it means that your code can both push and pull data along the same pipe. In one scenario a client could pass binary data like a graphic file to a component via one or more pushes and then pick up the processed data via one or more pulls.

What about the asynchronous versions of the pipe interfaces? Well, you can use these to free the data consumer thread from blocking (which is a good thing if the client thread is a UI thread) and to make better use of the threads in the data source. When pulling data via the async versions of the pipe interfaces the data consumer thread initiates the transfer by calling the Begin_Pull() method indicating how many items are required. It can then perform some other processing and return later to obtain the data (and the number of items returned) by calling the Finish_Pull() method. During this time COM will have received the data and cached it ready for collection.

The only oddity that I can see with this asynchronous method is that when you call Finish_Pull() you pass the buffer which you want filled, after the actual transfer has been performed. Presumably COM will have read the data into another buffer and when you call Finish_Pull() it will copy the data from its buffer into yours.

The asynchronous version of Push() is useful to send data to another process without the current thread blocking, which is quite a useful thing to do especially if the amount of data is large. Since there are no [out] parameters on Push() it means that the only purpose of calling Push_Finish() is to allow COM to clean up any resources it may have used, and to determine if the push was successful (the return value from Push_Begin() only indicates that the method call was accepted by COM).

Finally, one tip. To use pipes you must have the headers and libs from the most recent Platform SDK and also make sure that in your stdafx.h you define _WIN32_WINNT to have a value of 0x500.

What do you think of this article?

You can also write a review. We will publish the best ones here on this article. Send your review to [email protected].

Mail a question to the author!!

As part of the IDevResource commitment to Open Publishing, all of our authors are available to answer all of your trickiest questions at Author Central. For information about the authors, or to mail a question, visit them at Author Central.

Want to read more articles by this author?

Try these:

Byte size articles:

ATL Server By Richard Grimes, 220200
Richard's report from European Devweek 2000, and VCDC Bonn 2000 gives an insight into the new ATL Server technology.
Go To Article.

COM and Apartments By Richard Grimes, January 7th 2000
A discussion on the creation and use of "Apartments" in Win32.
Go To Article.

An Introduction to WTL By Richard Grimes, January 7th 2000
Richard gives a basic introduction to WTL.
Go To Article.

An Introduction to Interface Programming By Richard Grimes, January 7th 2000
Richard describes the background to interfaces.
Go To Article.

Full size articles:

What is Async COM? By Richard Grimes.
Go To Article

Microsoft Transaction Server By Richard Grimes, December 9th 1999
Introduces Microsoft Transaction Server
Go To Article.

What COM is all about By Richard Grimes, December 6th 1999
An introductory article to COM, covering all basics including OLE, Activex and DLL's.
Go To Article.

More COM Library articles:

CAtlBitmapButton - ATL/WTL Ownerdraw Superclassed Bitmap Button By Amit Dey.
Amit describes building a simple user interface consisting of a series of bitmap buttons in a dialog.
Author: Amit Dey
Date Submitted: June 8 2001

ATL COM and ADO By Paddy Srinivas.
COM+ Events addresses many shortcomings of Connection points. Paddy Srinivas walks us through the COM+ Events System.
Author: Paddy Srinivas
Date Submitted: March 7 2001

ATL COM and ADO By Amit Dey.
New author Amit Dey (he's looking for a job BTW ;-) ) explains his recent experiences with ATL COM and ADO. He simply explains the fundamentals of ADO and then runs through some ATL code examples in his application.
Author: Amit Dey
Date Submitted: March 6 2001

COM Singletons: A Dangerous Animal By Richard Blewett.
Richard Blewitt simply explains the use of COM Singletons.
Author: Richard Blewett
Date Submitted: January 24 2001

ASP COM Objects By Jan Verhoeven.
Creating ASP com objects, or Active Server Objects with Delphi 5 is very easy, once you know how to do it.
Author: Jan Verhoeven
Date Submitted: November 1 2000

SafeArrays - For the Beginner By A. Abdul Azeez.
This article is a primer to Safe Arrays and can be used by any beginner to Safe Arrays.
Author: A. Abdul Azeez
Date Submitted: November 1 2000

COM+ Basics - Creating your first COM+ Application By Martin Lapierre.
Author: Martin Lapierre
Date Submitted: June 1 2000

COM+ - the backbone of Windows DNA By Mahesh Bhide.
Author: Mahesh Bhide
Date Submitted: June 1 2000

Exploring COM Threading and Apartments By Anthony Toivonen.
Anthony Toivonen wants you to figure it out for yourself: he guarantees success in threads and apartments after reading this article.
Author: Anthony Toivonen
Date Submitted: May 17th 2000

COM on Linux By Frank Rem.
A description of how to code a DCOM client for Linux without using Microsoft products.
Author: Frank Rem
Date Submitted: May 17th 2000

ATL Server By Richard Grimes.
A description of ATL Server.
Author: Richard Grimes
Date Submitted: May 9th 2000

ATL Internals - Part 2 By Shivesh Viswanathan.
Following on from Shivesh's first article of this two part series, this article covers the details of the internals of ATL.
Author: Shivesh Viswanathan
Date Submitted: May 3rd 2000

How to use DDX with WTL By Girish Bharadwaj.
Author: Girish Bharadwaj
Date Submitted: March 27th 2000

COM Patterns By Tony Toivonen.
Author: Tony Toivonen
Date Submitted: March 27th 2000

COM+ Object Pooling By Jeremiah Talkar.
Author: Jeremiah Talkar
Date Submitted: March 23rd 2000

What are COM Pipes? By Richard Grimes.
Author: Richard Grimes
Date Submitted: March 23rd 2000

Java COM Integration - Use Visual J++ to implement COM Objects By Gopalan Suresh Raj.
Author: Gopalan Suresh Raj
Date Submitted: March 9th 2000

Developing an MSMQ Server Application using VJ++ By Gopalan Suresh Raj.
Author: Gopalan Suresh Raj
Date Submitted: March 9th 2000

Developing an MSMQ Client using VJ++ By Gopalan Suresh Raj.
Author: Gopalan Suresh Raj
Date Submitted: March 9th 2000

What is Async COM? By Richard Grimes.
Author: Richard Grimes
Date Submitted: March 8th 2000

Coding a DCOM Server Component from IDL By Gopalan Suresh Raj.
Author: Gopalan Suresh Raj
Date Submitted: February 10th 2000
Level of Difficulty: Advanced
Subjects Covered: IDL, DCOM, J++
Pre-required Reading: Component Object Model

Coding a DCOM Client By Gopalan Suresh Raj.
Author: Gopalan Suresh Raj
Date Submitted: February 10th 2000
Level of Difficulty: Advanced
Subjects Covered: IDL, DCOM, J++
Pre-required Reading: Component Object Model, Coding a DCOM Server Component from IDL

Threads and Apartments By Brad Wilson.
Brad's article gives detailed information about apartments and their relationship to threading and synchronization. The goal is to demystify what is a very important, yet under-documented system in COM.
Author: Brad Wilson
Date Submitted: Feb 04 2000
Level of Difficulty: Intermediate
Subjects Covered: Single Threaded Apartments, Multi Threaded Apartments, Apartments.
Pre-required Reading:

WTL Architecture By Richard Grimes.
This article covers the basics of the WTL architecture, it describes the types of applications that you can create and how WTL manages threads. The example monitors the debug stream and uses WTL to present this data.
Author: Richard Grimes
Date Submitted: Feb 02 2000
Level of Difficulty: Intermediate
Subjects Covered: WTL applications architecture, WTL UI update, OutputDebugString and debugging API
Pre-required Reading: assumes a knowledge of ATL and of Win32 application development (for example "Professional ATL COM Programming", Richard Grimes, Wrox Press)

The MTS Series by Gopalan Suresh Raj:

Microsoft Transaction Server By Gopalan Suresh Raj.
Gopalans introductory article on Microsoft Transaction Server introduces the basics to MTS, and leads in to the example articles included in the series.
Author: Gopalan Suresh Raj
Date Submitted: Jan 26 2000
Level of Difficulty: Beginners / Intermediate
Subjects Covered: MTS
Pre-required Reading: None

Developing a Simple MTS Server Component By Gopalan Suresh Raj.
Part 1 of a two part example.
Author: Gopalan Suresh Raj
Date Submitted: Jan 26 2000
Level of Difficulty: Beginners / Intermediate
Subjects Covered: MTS
Pre-required Reading: Microsoft Transaction Server

Developing a Simple MTS Client Application By Gopalan Suresh Raj.
Part 2 of a two part example.
Author: Gopalan Suresh Raj
Date Submitted: Jan 26 2000
Level of Difficulty: Beginners / Intermediate
Subjects Covered: MTS
Pre-required Reading: Microsoft Transaction Server, Developing a Simple MTS Server Component

Developing The Bank Account IDL By Gopalan Suresh Raj.
A Three-Tier Architecture for a Bank Checking Account - Developing The Bank Account IDL is part 1 of a 3 part example.
Author: Gopalan Suresh Raj
Date Submitted: Jan 26 2000
Level of Difficulty: Beginners / Intermediate
Subjects Covered: MTS
Pre-required Reading: Microsoft Transaction Server

MTS Server Component By Gopalan Suresh Raj.
A Three-Tier Architecture for a Bank Checking Account - MTS Server Component is the second part of this three part example.
Author: Gopalan Suresh Raj
Date Submitted: Jan 26 2000
Level of Difficulty: Beginners / Intermediate
Subjects Covered: MTS
Pre-required Reading: Microsoft Transaction Server, Developing The Bank Account IDL

MTS Client By Gopalan Suresh Raj.
A Three-Tier Architecture for a Bank Checking Account - MTS Server Component is the third part of this three part example.
Author: Gopalan Suresh Raj
Date Submitted: Jan 26 2000
Level of Difficulty: Beginners / Intermediate
Subjects Covered: MTS
Pre-required Reading: Microsoft Transaction Server, Developing The Bank Account IDL, MTS Server Component