In, Out, In-Out, Make Up Your Mind Already

I was talking about reference types vs. by-reference variables a while back. Recall that both JavaScript and VBScript have reference types (“objects”) but JavaScript does not have by-reference variables.

COM supports passing variable references around, but unfortunately the intersection of early-bound COM and late-bound IDispatch is a little bit goofy. There are a few problems that you need to be aware of when you’re trying to get VBScript code to talk to COM objects that expect to be passed references to variables. (Obviously attempting to do so from JavaScript is just a non-starter, as JavaScript will not pass variable references at all.)

The most common problem happens when you are trying to call a method on a dispinterface where the vtable entry for the method looks something like this

HRESULT MyFunction([in, out] BSTR * pbstrBlah );

Calling this method via VBScript like this produces a type mismatch error:

Dim MyString
MyString = "foo"
MyObj.MyFunction MyString

And calling it like this does not produce an error but also does not actually pass the value by reference:

Dim MyString
MyString = "foo"
MyObj.MyFunction CStr(MyString)

The latter behaviour is completely expected — what you’re passing is the output of a function, not a reference to a variable. There is no reference available, so no value is filled in. But why does the former fail?

Well, VBScript does not know what the callee is expecting as far as types go. That’s what “late bound” means — the callee has to do the work of determining how to suck the relevant data out of the variants passed to Invoke, and the callee has to somehow call the underlying vtable with the correct types. So VBScript sees

MyObj.MyFunction MyString

and passes a reference to a variant. All variables are variants in VBScript.

Why does VBScript produce a type mismatch error here? VBScript doesn’t! The object produces the type mismatch error, which VBScript dutifully reports. The object’s implementation of Invoke calls the default implementation of Invoke provided for you by the type library implementation. That thing says “I’ve got a reference to a variant, and that variant is a string. I need a reference to a string. That’s a type mismatch.”

This seems like a missed trick; if I were designing such a system, I’d put some additional smarts into it that would handle this case. Clearly from a reference to a variant that contains a string one can obtain a reference to a string — just take the address of the string field of the variant! However, for reasons which are lost to the mists of time, the default implementation does not do that.

My advice to you would therefore be

  • If you want to write an object that can be easily used from script, do not have any [in, out] parameters, because JScript can’t pass references.
  • If you must have in/out parameters, make them variants, because VBScript can’t pass any other kind of reference.
  • If you must have nonvariant in/out parameters, write some fixer-upper code for your IDispatch implementation which transforms byref variants pointing to strings into byref strings. (Or whatever byref type you require.) But if you do that, make sure you get it right.
  • Do not attempt to write your own IDispatch implementation, as there are many pitfalls (which I may discuss at another time).

That’s the most common problem I see. The other common problem involves out parameters which are not in/out or out/retval parameters. Just-plain-out parameters cause memory leaks. Consider our earlier example:

Dim MyString
MyString = "foo"
MyObj.MyFunction MyString

Suppose MyFunction takes an in/out variant, and fills in the byref variant with the string “bar”. The implementor expects that something will come in and something will go out, and the rule is that the callee frees the coming-in memory before replacing it with the going-out memory. The caller is then responsible for freeing the going-out value.

But if MyFunction takes a just-plain-out variant then the callee does not free the incoming memory. It assumes that the incoming memory is garbage because it has specifically been told that nothing is coming in.

How does VBScript know whether the callee is in-out or out? VBScript doesn’t! Knowing that requires either compile time knowledge or a very expensive run-time lookup (the results of which are difficult to cache for the same reasons that dispatch ids are difficult to cache.)

The practical result is that if you pass an object or string to a callee expecting an out variant, the string or object pointer will be overwritten without first being freed, and the memory will leak. You should ensure that you always pass empty variants if you must call an object method that takes an out parameter. Yes, this is a violation of the design principle that late-bound calls must have the same semantics as early bound calls, but unfortunately, that’s just the way OLE Automation is and there’s nothing we can do about it now.


Commentary from 2019

Most of the responses to this article were of the form “I wish this had been documented earlier”. Indeed, that is the primary reason why I started this blog! I felt that a lot of this stuff was under-documented in the 1990s, and I belatedly wanted to fix that for posterity.

Now that posterity has arrived, we can see that the design of IDispatch had a number of flaws. An interface specifically designed for “late binding” does not need to provide a mechanism for callees to interrogate callables to determine the types expected, though it sure would be nice to have. But the fact that the caller needs to know the calling convention in order to avoid memory leaks, and there is no cheap way for the caller to interrogate the object to find out what it expects, is terrible! It forces the script engine developer to trade off between bad performance on the one hand, and a memory leak on the other.

This also illustrates one of the costs of runtimes that lack garbage collection; think about how much effort have we put into designing and implementing memory management protocols in COM APIs. None of that work had to be done for C# APIs, freeing the developer to concentrate on the business semantics rather than the storage mechanisms.

1 thought on “In, Out, In-Out, Make Up Your Mind Already

  1. You cannot call `MyObj.MyFunction MyString` with `MyString` being a Variant w/ early-binding too in VB6. I think there is nothing goofy in `IDispatch` per se in this case as it just emulates 100% the *compile-time* error when early-bound call is attempted.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s