A number of people have pointed out to me over the years that VBScript’s Round function is a bit weird. It seems like it should be pretty straightforward — you pick the integer closest to the number you’ve got, end of story. But what about, say, 1.5? There are two closest integers. Do you go up or down?

The Round function goes to the nearest integer, and if there are two nearest integers then it goes to the even one. 1.5 rounds to 2, 0.5 rounds to 0.

Why’s that? Why not just arbitrarily say that we always round down in this situation? Why round down sometimes and up some other times? There actually is a good reason!

This algorithm is called the Bankers’ Rounding algorithm because, unsurprisingly, it’s (allegedly) used by bankers. Suppose a data source provides data which is often in exactly split quantities — half dollars, half cents, half shares, whatever — but they wish to provide rounded-off quantities. Suppose further that a data consumer is going to derive summary statistics from the rounded data — an average, say.

Ideally when you are taking an average you want to take an average of the raw data with as much precision as you can get. But in the real world we often have to take averages of data which has lost some precision. In such a situation the Banker’s Rounding algorithm produces better results because it does not bias half-quantities consistently down or consistently up. It assumes that on average, an equal number of half-quantities will be rounded up as down, and the errors will cancel out.

If you don’t believe me, try it. Generate a random list of numbers that end in 0.5, round them off, and average them. You’ll find that Bankers’ Rounding gives you closer results to the real average than “always round down” averaging.

The Round, CInt and CLng functions in VBScript all use the Banker’s Rounding algorithm.

There are two other VBScript functions which turn floats into integers. The Int function gives you the first integer less than or equal to its input, and the Fix function gives you the first integer closer to zero or equal to its input. These functions do not round to the nearest integer at all, they simply truncate the fractional part.

This article generated a lot of feedback pointing out:

• Format and FormatNumber do not use Bankers’ Rounding.
• A great resource for people trying to understand how pre-.NET Visual Basic proper does rounding is here
• Bankers’ Rounding has an additional nice property that I did not mention. Suppose we have a principal payment of $123.755 and an interest payment of $1.245.  The true sum is $125.00. If we round off the quantities first, then we get $123.76 and $1.24, and the sum is still $125.00. Other rounding algorithms would produce an error of a penny.
• Bankers’ Rounding also has nice properties when dealing with the sums of quantities that have been computed by a round-then-subtract operation.
• There was some debate over whether Bankers’ Rounding is “fair” because there are five possible odd digits: 1, 3, 5, 7 and 9, but only four possible even digits, 2, 4, 6 and 8.  I leave it as an exercise to the reader to determine whether this objection has merit. (The commenter later returned to say that what they meant was that the distribution of odd and even digits was not necessarily uniform.)
• The ROUND function in Excel may not use Bankers’ Rounding; there was some confusion on this point and I have never bothered to actually check.
• Bankers do not actually use Bankers’ Rounding, apparently; rounding ties “upwards” is the standard method. I do not know if “upwards” means “towards positive infinity” or “away from zero”, since those are different for negative numbers.

There was also an irritating back-and-forth with readers who believed they had found “flaws” in the tie-breaking algorithm; in every case, the algorithm was not tie-breaking because there was a clearly “closer” value, and that was the value chosen.

Thank goodness, the .NET Round method takes an argument so you can say what kind of rounding you want, rather than hoping for the best.

I said earlier that certificates could be used to establish identity and hence the right
to run trustworthy software on your machine. I want to emphasize that this is not all that certificates are used for.

A lot of people are confused by this, including a lot of very smart, technically savvy developers. Cryptographic security is complex and poorly understood.

Let me give you an example. One time I went to a friend’s web site. My friend is a smart and highly experienced developer. His web site happens to use HTTPS, the “secure protocol”, and I was very surprised when I got an error message from IE saying that the certificate associated with the site could not be trusted because it was self-signed. In other words, the certificate says “The bearer’s name is Sven Svenson. The bearer’s identity has been confirmed by Sven Svenson.”

Obviously that is not any kind of identity-establishing evidence! To trust the cert you have to trust the certifying authority, but the certifying authority is the person whose identity is in question! This chain of trust is not rooted in an explicitly trusted authority, so it is worthless. What is stopping, say, Bjorn Bjornson from typing up the same certificate? Nothing!

I asked my friend why he didn’t use a Verisign certificate and his answer surprised me: “Nobody’s told me that they won’t use the service unless I got a Verisign cert. And if they did, I’d probably tell them that they shouldn’t use my system if they don’t trust me.”

My friend was confusing code signing with secure HTTP. HTTPS does not use certs to establish a trust relationship between the client and the server by establishing the identity of the author! HTTPS uses certs to establish secure communication over an insecure network by establishing identity of the web site.

Those sure sound similar but they are in fact completely different uses of certs. Why? Because in the first case, it is the code author who is (potentially) untrusted. In the second case, it is the network that is (definitely) untrusted.

Before I continue let me briefly explain what the goal of HTTPS is, and how it works.

The goal of HTTPS is to ensure that when you send information over the internet, two things happen. First, the information is encrypted so that if it is intercepted, the eavesdropper cannot read it. Second, the server that you’re talking to really is the server that you think you’re talking to.

We have these two goals because there are two threats: first, someone might be listening in on the traffic between the client and the server, so you need to ensure that they can’t make sense of what they overhear. Second, someone may have set up a “fake” server that fools your client into believing that it is talking to the “real” server, so you need to ensure that you are talking to the “real” server.

When you go to a secure web site — like your bank, or Amazon or some other web site involving transmission of credit card numbers over the internet — the first thing the web site does is send you a certificate containing the name of the web site. The certificate has been signed by a trusted root — Verisign, for example. Your browser can then compare the certificate’s web site name with the web site you actually went to, and because it is vouched for by Verisign, you know that you really are talking to the right server. The certificate also contains information about how to encrypt messages so that only the server can decrypt them. That is then enough information to mitigate both threats and establish a secure channel. (The details of how that channel is established are not particularly germane; I may discuss this in a future post.)

I’ll say it again: the certificate is not there to establish a trust relationship between the client and the server. The certificate is there so that you know that you are talking to the real server, and no one can eavesdrop. This is the internet we’re talking about: for all you know, evil geniuses have cut every line between you and the entire internet and have inserted their own evil routers and servers that look just like the real internet. (For all you know, this is someone else’s blog!) It’s Verisign’s job to ensure that those evil geniuses never get their hands on a Foo.com certificate vouched for by Verisign, because that is the only evidence you have that you are actually talking to the real Foo.com.

So when you go to a web site with a self-signed HTTPS certificate, IE pops up an “abort
or continue?” dialog box. What that box is trying to communicate is:

“You are trying to communicate securely with FooCorp’s web server but because the certificate does not establish a chain of trust, IE is unable to determine if the insecure internet between you and FooCorp has been taken over by evil genius hackers who are presently spoofing you into believing that you are on the real internet. Do you want to go ahead and possibly be spoofed, or stop right now?”

Unfortunately, the dialog box is not particularly clear on this point, and besides, everyone clicks “OK” without reading dialog boxes. But that’s another story.

To sum up: whether you trust FooCorp or not is not the issue — if you’re going to send them your credit card number, presumably you already trust them! What you don’t trust is the wiring between you and them. There might be hackers at their ISP, or your ISP, or at any node on the network. There might be people listening with packet sniffers and packet replacers. There might be people listening to your wireless network traffic. HTTPS is designed to protect users against untrustworthy, insecure or hostile network providers by ensuring identity and encrypting traffic.

There were a lot of good responses to this article, mostly on the topic of the correct uses of self-signed certificates, and the business model and trustworthiness of VeriSign.

There are of course valid uses of self-signed certificates; trust chains typically end in a self-signed certificate after all. The point is that you must have some tamperproof way of obtaining a self-signed root and some way of verifying the identity associated with that certificate; obviously neither transport nor verification must involve checking the signature of the certificate! But if you have some other way to do those things, then a self-signed cert is fine; put it in your root store and you’re all set.

Self-signed certs also solve the problem of “I don’t care about the identity of who I’m talking to, but I do care that this identity remains constant over time”. Whether that is a realistic scenario or not, I can’t say.

VeriSign at the time charged $1000 for a certificate, which many people noted was high for individuals, though perhaps not for corporations. Theses days of course there are free certifying authorities.

The suggestion was made that “loop signed” certificates could be a thing; you could have a group of certifying authorities get together and X signs Y’s certificate, who signs Z’s certificate, who signs X’s certificate, and this is in a sense stronger than a self-signed certificate. I don’t know; that sounds complicated and like it doesn’t actually solve the problem, but I have not done a detailed analysis.

People also pointed out that VeriSign had made some high-profile mistakes: granting a Microsoft certificate to someone who didn’t work for Microsoft, messing up DNS, and so on.

One day Peter Torr and I walked into Herman Venter’s office (Herman was the architect and primary implementer of JScript.NET). We were grinning. Herman knew what that meant. He took one look at us and said “Uh oh, here come the Evil Security Twins.” And indeed, we had found another potential vulnerability in JScript .NET — fortunately, before we shipped.

My Evil Security Twin has an interesting post today about the need to trust both code
and the code container. I thought I might discuss the semantics of some of the words Peter uses. Security experts often use word pairs such as safe/dangerous, trusted/untrusted, and benign/hostile without clearly distinguishing them from each other. These distinctions are both important and frequently misunderstood. Essentially the differences are these:

A particular assembly is safe if it is unable to harm you: modify your disk, steal your data, deny service, and so on. The safety or dangerousness of an assembly is fundamentally a technical question: what does the assembly attempt to do? For instance, does it read the disk? Then it might steal secrets. Does it create dialog boxes? Then it might create hundreds of them and make it hard to use your machine.

The level of trust you assign to an assembly is essentially the set of permissions you are willing to grant it based on evidence. For instance, if you believe that if an assembly from the Internet is granted permission to access your disk then it might misuse that ability to do you harm, you should not grant the permission.

Administrators express their opinions about trust by configuring their policies appropriately. “Do not trust code from the Internet to access the disk” is an example of such a policy.

Whether an assembly is hostile or benign depends on the mindset and goals of the assembly’s author. An assembly that creates a file might be perfectly benign, or it might be creating the file in order to consume all the hard disk space on your machine as a denial-of-service attack. An assembly that creates a dialog box that looks just like the Windows password dialog might be attempting to steal your password.

Unfortunately, there is no way to detect the intent of the programmer. If there were then security would be easy: use your magical ESP powers to see what the programmer was thinking and then just prevent code written by hostile people who hate you from running! In the real, non-magical world all that the .NET security system can do is restrict the abilities of an assembly based on the available evidence.

Notice that essentially we use these words technically in the same way we do in everyday speech. A tennis ball is inherently quite safe and an axe is inherently quite dangerous. You trust that the axe maker made a trustworthy product that accurately advertises its dangerous nature (rather than masquerading as a safe object which you could then accidentally misuse).

Whether someone carrying an axe is hostile or benign has nothing to do with the axe, but everything to do with their intentions. And whether you trust that person to do yard work is a question of what evidence you have that they’re trustworthy. A policy that codifies this trust decision might be “Don’t give axes to unidentified strangers.”

When we get into cryptographic evidence, things get even more confusing. What are all of these certificates for, and who signed them, and why do we trust them? We can continue our analogy.

Some guy shows up at your door to do the yard work. You’re considering handing him an axe. What evidence do you have that this is a wise thing to do?

You could start by asking for some ID. The guy hands you a card that says “Hi, my name is Sven the Lumberjack”. Does this establish identity? Obviously not. Does it establish trustworthiness? No! So what’s missing?

What’s missing is the imprimatur of a trusted authority. If instead Sven hands you a driver’s license with a photo on it that matches him and a name that matches the name he claims is his, then that’s pretty good evidence of identity. Why? Because you have a trust relationship with the entity that issued the document! You know that the department of transportation doesn’t give a driver’s license out to just anyone, and that they ensure that the picture and the name match each other correctly.

So far we’ve established identity but have we established trustworthiness? Maybe, maybe not. Identity is a crisp, technical question. Trustworthiness is a squishy, human question. Maybe you’re willing to say that anyone who is willing to identify themselves is trustworthy. Maybe you demand extra credentials, like a lumberjack union membership card — credentials that certify relevant abilities, not just identity. Maybe you’re worried about fake ID’s.

All that is up to you — you’re the one handing this guy an axe. How paranoid are you? Set your policies appropriately. Some of us are more paranoid than others, like Peter “I only read ASCII email” Torr. That’s hard-core, dude.

But I digress. My point is that we construct chains of trust. Every link in that chain must hold up, and the chain has to end in some authority which you explicitly trust.

In the computer world, your machine has a certificate store containing a list of “trusted roots“, like Verisign. By putting Versign in the trusted root store, you are saying “I explicitly trust Verisign to do their job of issuing and revoking credentials that establish identity and trustworthiness.”

If Sven goes to Verisign and asks for a “code signing” cert, Verisign will confirm his identity and issue a certificate that says “this guy is known to Verisign and this certificate can be used to sign executable code.” It’s a driver’s license for coding.

When you download some code off the internet, the security system checks to see if you trust it — does it come from someone with an identity verified by Verisign? Do you trust that individual? And does the certificate they’re showing you say that they have the right to sign executable code? If so, then the code runs. If not, then the chain of trust is broken somewhere, and it doesn’t run.

This article had some good reader feedback. Two highlights:

First off, a reader noted that good security systems get the transitivity relationships correct. If I trust Alice, and Alice trusts Bob, that does not necessarily mean that I trust Bob.

Second, Peter pointed out that reading HTML-rendered email is a “horrible” user experience regardless of any security concerns; remember, it was 2003; there were real security concerns when reading email still!

As I mentioned before, Peter is still at Microsoft as of this reading. Oddly enough, Herman sits just a few desks away from me, and is once again working on JavaScript optimizers. Plus ça change.

While I was waiting for a shuttle the other day (Microsoft has a fleet of shuttle busses that take people all over the campuses) I was thinking about optimization heuristics.

Often we want to find an algorithm that determines the best way to do something. For example, the traveling salesman problem: given a set of cities and the cost of driving between all of them, what is the cheapest route that visits every city at least once? Such problems can be extremely difficult to solve! But often we miss out on the fact that we don’t need to find the optimal solution, we need to find the “good enough” solution. Or we need to find a solution that has bounded inoptimality. Often finding the best solution is impractical, but it would be nice to know that you can find a solution that doesn’t suck  too bad.

For the traveling salesman problem, for instance, you can easily construct the minimum spanning tree of the graph. Then the route becomes trivial: pick any node as the root, do a depth-first traversal of the spanning tree and go back to the root. You’ll visit every edge twice and end up where you started from. The total cost is twice the cost of traversing the spanning tree once, and obviously the cost of the optimal solution can’t possibly be better than the cost of traversing the minimum spanning tree once.

This doesn’t give you an optimal solution (unless your graph is already a tree, of course) but it does give you a solution that is guaranteed to be between 100% and 200% of the optimal solution, and that might be good enough.

This sort of 200% heuristic crops up all the time. Do you rent DVDs or buy them? Suppose a DVD costs $24 to buy and $6 to rent. Obviously the optimal solution is to buy it if you are going to watch it four or more times, rent otherwise — but that requires perfect knowledge of the future.

Can we come up with an algorithm that minimizes the maximum suckiness, given no information about the future? Well, the worst possible situation is buying a DVD and watching it once — you just overpaid by a factor of four. Actually, no, a worse situation still is renting the DVD so many times that you end up paying way more than the purchase price.

The solution? Rent the DVD the first four times, and then the fifth time, buy it. No matter how many times you watch it, the worst you can do is pay 200% of the optimal price and typically you’ll do a lot better. (Particularly if you can bring to bear more information, such as being able to forecast the number of future viewings.)

This same heuristic applies to waiting for shuttles, which is why I was thinking about this in the first place. How long should you wait before you give up and walk? If the shuttle is going to be right there 30 seconds after you call, it’s foolish to walk. But it is also foolish to wait ten times as long as it would have taken you to walk.

I waited as long as it would have taken me to walk, and then I walked. Fortunately it was a nice day yesterday.

I noticed immediately after posting this in 2003 that I got the algorithm wrong! It would be better to buy the DVD on the fourth viewing. Then the worst case is 4 viewings for $42, which is less than twice the optimal case. But whatever, you take my point I’m sure.

Don’t forget people, my degree is in mathematics, not arithmetic.

As I write this update in 2019 it has been many years since I last rented a DVD, so the scenario is no longer topical, but I hope the general point is still of interest. And I really hope that Scarecrow Video, the greatest video store in the world, does not go out of business — but I just have no incentive to go there anymore.

Readers pointed out that in many scenarios the parameters of the problem are known at compile time, and that they are willing to solve the problem once and bake the solution into the program. For example, optimize route-finding for AI characters in games doesn’t typically depend on details that change during play, and so can be pre-computed with long-running, more accurate algorithms, rather than approximated at runtime with faster algorithms.

Another reader pointed out that attempts to predict the future are best modeled as probabilistic algorithms, and those algorithms can consume evidence (what do reviewers think? How correlated are my opinions with reviewers? What did people who also bought this also buy that I bought? and so on) to compute the posterior probability that I’ll watch it again. At the time I am porting this article over I am in the middle of my “Fixing Random” series on probabilistic programming in C#, so that’s been on my mind.

My office nameplate was entirely apropos; LoadPicture continued to plague me throughout my career. During the Windows Security Push last year we finally turned it off. When VBScript is running in IE, the LoadPicture method causes an illegal function call exception.

Why’s that? There’s a big security hole in LoadPicture. Not because its buggy, but because it works too well!

When I added LoadPicture to VBScript I considered the security implications, but did I mention this was in my second week at Microsoft? I knew very little about security then. I reasoned that sure, this is a method that can access your hard disk, but what’s the worst it can do? It can open a locally stored picture file. Big deal! It can’t write any information back to the disk, so the risk seems very low.

Now of course we use the formal threat modeling process that my colleagues Mike Howard and David Leblanc describe in their book Writing Secure Code. We brainstorm possible threats and vulnerabilities, document them, and ensure that the code is built to resist those threats and mitigate those vulnerabilities. But in the bad old days the kind of off-the-cuff informal analysis I’ve just described was pretty typical.

The vulnerability in LoadPicture is that an attacker can write a web page which, when you visit the page, reports back to the web server your user name, what programs you have installed and potentially the names of your files. This is an information leaking vulnerability.

It can’t actually harm you directly, but attackers want to know everything about your machine. Knowing what all the user names are is a good first step in guessing their passwords. Knowing what programs you have installed tells them whether you’ve installed the latest patches, and hence whether you’re a potential target. And of course there is also the privacy angle. It isn’t anybody else’s business whether you have Office installed or not!

The vulnerability exists because, ironically, we did too good a job of reporting error messages. LoadPicture succeeds if you give it a valid path to an image file. But if you give it an invalid path, it gives you different error messages for “that’s not a legal path“, “that’s a legal path to a file which is not an image“, and “that’s a legal path, but it’s a directory, not a file“.

An attacker can make guesses about where various files and directories are on your system. The “Documents and Settings” folder contains a directory for every user on the machine. Thus the user names could be discovered via a dictionary attack (or brute-force attack; user names are usually short).

Of course, it could be worse. There was a bug in early versions of the CLR (which I believe was fixed before the first beta shipped, fortunately) where you could get an error message something like

Path discovery security exception: You are not allowed to determine the name of directory ‘c\:foobar’

Super! Thanks for letting me know!

The moral of the story is that as developers we tend to design code that produces the error messages that we need to be able to understand the problem and fix it. Unfortunately, those are the same data that attackers need to understand the system to break it. We’ve got to be careful not to leak information in error messages to insecure callers.

As promised, my least customer impactful bug ever.

VBScript version 1.0 was written by one of the VB compiler devs, one of those uber-productive developers who implements compilers on the weekend for fun. It was implemented and tested extremely rapidly, and as a result, a few of the less important methods in the VB runtime were not ported to the VBScript runtime. My first task as a full timer at Microsoft was to add the missing functions to VBScript 2.0.

On August 5th, 1996, I implemented LoadPicture, which, as you might imagine, extracts a picture from a storage. Here’s a scrap of the code I wrote:

IDispatch * pdisp;
// [... open storage and stream ...]
hresult = OleLoadPicture( 
  pstream, 0, TRUE, IID_IPicture, (void **)&pdisp );

Did I mention that I’d been writing COM code for all of two weeks at the time?

There’s a boneheaded bug there. I’m asking for an IPicture* and assigning the result to an IDispatch* This is going to crash and burn the moment anyone tries to call any method on that picture object because the vtable is going to be completely horked. I’ve violated one of the Fundamental Rules of COM. The fact that this bad code shipped to customers indicates that:

• Most seriously, I did not adequately test it before I checked it in
• my mentors did not adequately review the code
• the test team did not adequately test it before it shipped to customers

Those are all bad, and I am happy to say that today we are much, much more hard-core about peer-reviewing, self-testing, buddy-testing, and tester-testing code before it goes to customers than we were seven years ago.

But there is a silver lining of a sort — obviously no one at Microsoft bothered to run this code after I wrote it. But neither did any users. We didn’t get a bug report on this thing until February of 1998. This thing was in the wild for almost a year and a half before someone noticed that it was completely, utterly broken! No one really cared.

The next day the name plate on my office door said LoadPicture Lippert. Ha ha ha, very funny guys.

A few germane questions from the comments:

Is there any use for that thing?

You can’t use it in a web page that loads from the internet for security reasons. HTML-based applications that run locally have better mechanisms for loading pictures. There’s no reason to use it from Windows Script Host or Active Server Pages. The only real use is when you have a trusted script and it is scripting an ActiveX control that can have pictures added to it dynamically.

It’s ironic that this problem would have been avoided had you used your hated smart pointers. Do you know that there are no other places in the script engines where a CoCreateInstance creates an interface and the target pointer does not match?

There is exactly one place in the original script engine code calls CoCreateInstance and it gets it correct; the script engine would crash immediately if that weren’t the case.

Sure, smart pointers would have saved me in this case, but a technique that detects a problem at compile time that would have been caught immediately the first time I ran the code is not really that interesting! The problem here is that I was so unprofessional as to not even run the code I’d just written, and to not show it to anyone who would have immediately realized that it looked wrong.

My distaste for smart pointers is because the bugs that smart pointers cause are deep, insidious, hard to debug, and require detailed knowledge of both the underlying COM concepts that they are abstracting AND detailed knowledge of the smart pointer implementation.

Like I said, I’m not smart enough to use smart pointers. IUnknown::Release I understand, and when I screw up some refcounting, I know how to fix it quickly. Smart pointers work great most of the time, and when they fail, it takes me all day to track down the problem.

The other day I mentioned my worst customer-impacting mistake ever — marking a heavily used object with the wrong threading model. A number of people commented to me that it was unusual to see a developer own up to such a mistake in a public forum. (Surprisingly, no one pointed out that it was also odd to do so while talking like a pirate.)

Well, it’s because of restaurants, believe it or not.

My father has been in the restaurant business for many years. Something he taught me at an early age is that one measure of the quality of a restaurant is how few mistakes they make, but a more important measure is how they treat the customer once a mistake has been made. Do they apologize, take responsibility, and immediately act to correct the mistake, or do they engage in cover-ups, blame-shifting and foot-dragging? I don’t go back to the second kind of restaurant, no matter how good the food is.

The software industry is no different. Here are four aspects to consider:

• Customer focus: Most importantly, when a customer-impacting mistake is made it is necessary to inform customers of the problem, take responsibility and if possible fix the problem, period. If that means they think I’m a bozo, I’m cool with that. Preventing customer pain is a whole lot more important than anyone’s opinion of me.
• Public image: Just because I don’t walk up to you and say “Hi, I’m Eric and I’ll be developing your application infrastructure and development tools this evening” doesn’t mean that said tools aren’t developed by humans with names and faces! Too often companies such as Microsoft are perceived as faceless monoliths, a few dozen six-story sea-foam-green glass boxes that mysteriously transform money into code. This misperception belies the reality; I work with dedicated and talented people who care deeply about making their customers more productive. All those people — all the testers, developers, managers, writers, etc, — deserve credit for shipping great software. And when someone really screws up, well, adult humans should be capable of the occasional mea culpa.
• Evolution of the industry: The engineers who sign off on a bridge are quite literally putting their names on the line. If software engineering is ever going to evolve from a craft to a fully-fledged engineering discipline, we need to start acting like engineers. (And as a Waterloo math major, that’s hard for me to say with a straight face, believe me!) If I’m not comfortable with signing on the line and saying that this code is ready for prime time, then I shouldn’t be shipping it.
• Diffusion of knowledge: Finally, the best mistakes to learn from are other people’s mistakes. Learning from your own mistakes sucks, not to put too fine a point on it. If I can help other people learn from my mistakes, so much the better for the world as a whole.

Coming soon: Eric’s least customer-impacting mistake ever. But that will have to wait until this evening, because my band has a gig at our Product Unit meeting this afternoon. Yes, the Trinity Team has their own rock band! We are “Red Pills Reloaded“, aka “Rage Against The Blue Pills“.

There was some discussion in the comments about the $DATA stream vulnerability that had been recently discovered, but I don’t think I’ll summarize it here, aside to say that it was fundamentally a “fail to the insecure mode” error.

The Visual Studio Tools for Office team used code names that were all from the “Matrix” series of movies, and it was a fun theme. We put together a cover band — I played keyboards — and did a lot of Steely Dan and other 1970s era rock, which was a lot of fun. Choosing a name inspired by The Matrix was an obvious choice, and now I am intensely disappointed that “red pill” has since been co-opted by misogynist jerks; I can’t in good conscience wear my The Red Pills t-shirt in public.

I was discussing the difference between executing in local and global scopes the other day.  A reader points out something that I forgot to mention – there are two sneaky ways to manipulate the global namespace from an eval in JavaScript.

First, the Function constructor constructs a named function in the global scope. This had slipped my mind when I was writing the entry.

The second trick was very much on my mind but I did not mention as it would be yet another digression.  That is the fact that assigning a value to an undeclared variable creates a new variable in global scope.

This was on my mind because a couple weeks ago my friend CJ was debugging an irksome incompatibility between Gecko and IE. It turned out to hinge on the fact that in IE, fetching the value of an undefined variable is illegal, but setting it is legal.  According to CJ, in Gecko both are legal.

I wouldn’t know, never having actually used any browser other than IE since IE3 was
in development. (UPDATE: There is some dispute over this claim; like I said, I don’t know. Never tried it.)

If you look at the ECMAScript Revision 3 specification (E3) in some depth it becomes
clear that IE and Gecko are both in compliance with the spec, and yet incompatible
with each other.

“How’s that?” I hear you ask. The logic is a little tortuous!

Creating a new global variable when setting an undeclared variable must be legal according to E3 section 10.1.4, line 5, which states that an identifier undeclared in all scopes on the scope chain results in a “null reference”, and section 8.7.2, line 6 which states that an assignment to a null reference creates a new variable in the global scope.   IE does this, and I assume that Gecko does as well.

But setting the value of an undeclared variable must throw an error according to E3 section 8.7.1, line 3, which states that fetching the value of a null reference creates a ReferenceError exception. IE does this. If Gecko creates a variable in some scope rather than throwing a ReferenceError exception then clearly they have produced a situation in which a program running in Gecko has different semantics than when running in the browser used by the other 90% of the world.

Such situations are, as CJ, very painful for developers — mitigating this pain is why my colleagues and I went to the massive trouble and expense of defining the specification in the first place!  However, if that is the case then Gecko is not actually in violation of the specification thanks to E3 section 16, which states:

“An implementation may provide additional types, values, objects, properties, and functions beyond those described in this specification. This may cause constructs (such as looking up a variable in the global scope) to have implementation-defined behaviour instead of throwing an error (such as ReferenceError).”  [Emphasis added]

The E3 authors explicitly added the parenthetical clauses to make Gecko-like behaviour legal, though discouraged.  However, the clause is necessary — without this clause it becomes very difficult to define certain browser-object-model/script-engine interactions in a manner which does not (a) make both IE and Navigator technically noncompliant with the spec, (b) drag lots of extra-language browser semantics into the language specification and (c) make it difficult to extend the language in the future.

We earnestly wished to avoid all these situations, so the rule became “any error situation may legally have non-error semantics.” This is in marked contrast to, say, the ANSI C specification which rigidly defines what error messages a compliant implementation must produce under various circumstances.

As I noted above, this article produced immediate pushback from readers who said that CJ’s claim was wrong, that Gecko does the expected thing in this scenario, and suggested that he submit a reproducer of the defect.

Brendan Eich, the original designer of JavaScript and later CEO of Mozilla who stepped down after supporting anti-equality measures in California, said that I was “misreading” the specification, and that it was not intended to imply that any property can be referenced without error.

I have no idea what Brendan meant by this; the specification seems very clear to me; it specifically calls out that lookups in global scope need not throw.

All in all, the second half of this episode of FAIC seems to have been a waste of time and effort for all concerned.

A reader of the previous episode asked me

Why are there two types of multidimensional arrays? What is the difference between the arr(x)(y) and arr(x,y)notations?

Good question. There are two kinds of multidimensional arrays, called “rectangular” and “ragged“. (Or “jagged”; either is common.)

A rectangular array is, well, rectangular. In VBScript you say

Dim MyArray(3,2)

and you get an array with indices:

(0,0) (0,1) (0,2)
(1,0) (1,1) (1,2)
(2,0) (2,1) (2,2)
(3,0) (3,1) (3,2)

which makes a nice rectangle. A three-dimensional array makes a rectangular prism, and so on up into the higher dimensions.

Now, as I mentioned earlier, JavaScript does not have multidimensional arrays. A clever trick to simulate multidimensional arrays in JavaScript is to make an array of arrays:

var x = new Array(
  new Array(1, 2, 3),
  new Array(4, 5),
  new Array(6, 7, 8, 9));

Dereferencing the outer array gives you the inner array, which can then be dereferenced
itself:

print(x[2][0]); // 6

But you notice something about the indices if we write them out as before:

[0][0]  [0][1]  [0][2]
[1][0]  [1][1]
[2][0]  [2][1]  [2][2]   [2][3]

The indices make a ragged pattern, not a straight rectangular pattern.

You can have ragged higher dimensional arrays as well, though allocating all the sub-arrays gets to be a royal pain, and I recommend against it.

There are often times when you want ragged arrays even in a language that supports rectangular multi-dimensional arrays, so VBScript supports both. If you say

MyArray(2,3)

then you are talking to a rectangular two-dimensional array. If you say

MyArray(2)(3)

then you are talking to a one dimensional array that contains another one dimensional array.

While we’re on the subject of semantic differences between seemingly similar syntaxes, let me just take this opportunity to quickly answer a frequently asked question: why doesn’t for-in enumerate a collection?

A VB programmer is used to this printing out every item in a collection:

For Each Item In MyCollection
  Print Item
Next 

VB programmers introduced to JavaScript always make this mistake:

for (var item in myCollection)
  print(item);

and they are always surprised when this prints out a bunch of unexpected strings, or perhaps nothing at all.

The difference is quite simple.  In VBScript For Each enumerates the members of a collection. In JavaScript, for-in enumerates the properties of an object. In JavaScript if you have

var foo = new Object();
foo.bar = 123;
foo.baz = 456; 

then you can enumerate the properties:

for (var prop in foo)
  print (prop + " : " + foo[prop]) 

JavaScript needs such a control flow structure because it has expando objects and sparse arrays. You might not know all the properties of an object or members of an associative array. It’s not like VBScript where objects have fixed members and arrays are indexed by dense integer tuples. VBScript doesn’t need this ability, so it doesn’t have it.

Incidentally, in order to implement the for-in loop we needed to extend the functionality exposed by a dispatch object. Hence IDispatchEx which gives the caller the ability to enumerate dispids and go from dispid back to name.

In JScript to enumerate members of a collection, use the Enumerator object:

for (var enumerator = new Enumerator(myCollection) ; 
  !enumerator.atEnd(); 
  enumerator.moveNext())
{
  var item = enumerator.item();
  // ... 

The reaction I get from people who have not seen object-oriented enumerators before is usually “yuck!” This is an unfortunate reaction, as enumerator objects are extremely powerful. Unlike lexical For Each loops, enumerators are first-class objects. You can take out multiple enumerators on a collection, store them, pass them around, recycle them, all kinds of good stuff.

The semantics of the for in loop in JScript.NET are kind of a hodgepodge of both styles, with several interesting extensions. First off, if you pass an enumerator object itself to the JScript.NET for in loop, we enumerate it. If the argument is an object (or a primitive convertible to an object) then we enumerate its properties as JScript Classic does. If it is a CLR array, we enumerate its first dimension indices. If it is a collection, we fetch an enumerator and enumerate that. Otherwise, you’ve passed a non-enumerable object and we throw an exception.

There were a number of good responses to this article:

I like that you showed a common mistake of VBScript developers moving to JavaScript; what about VBScript developers moving to VB proper? What mistakes to they commonly make?

The advice I gave to the reader was to read Paul Vick’s blog, www.panopticoncentral.net I’m pleased that Paul is still blogging all these years later!

Here’s a clever trick for doing enumeration by passing in a function. We make a helper function, and then call it with the function to execute on each item:

function enumerate( coll, f )
{
  for( var en = new Enumerator(coll); 
    !en.atEnd(); 
    en.moveNext() )
  {
    f( coll.Item() );
  }
}
... 
enumerate(collection, function(item) { ... });

Though that is slick I am generally not that big a fan of mixing functional and imperative programming in cases where there’s an existing language construct that does the job.

I do sometimes need the ability to iterate property names in VBScript because it is often unclear what the properties of an object are; not all objects are well-documented.

The problem is that a lot of OLE Automation objects do not expose a mechanism for doing so; it can be tricky to go from an instance of an object to its type information.

Why are VBScript and JavaScript different in this regard?