I’m going to take a quick intermission from talking about the type system, but we’ll pick it up again soon.  I’ve been thinking a lot lately about philosophical and practical issues of thin client vs. rich client development.  Thus, I ought to first define what I mean by “thin client” and “rich client”. 

Theory

We used to think of Windows application software as being shrink-wrapped boxes containing monolithic applications which maybe worked together, or maybe were “standalone”, but once you bought the software, it was static — it didn’t run around the internet grabbing more code.  If the application required buff processing power and lots of support libraries in the operating system, well, you needed to have that stuff on the client.  The Windows application model required that the client have a rich feature set.

This is in marked contrast to the traditional Unix model of application software.  In this model the software lives on the server and a bunch of “thin” clients use the server.  The clients may have only a minimal level of memory and processing power — perhaps only the ability to display text on a screen!

The ongoing explosion of massive interconnection via the Internet starting in the 1990’s naturally led software developers to rethink the monolithic application model.  Multi-tiered development was the result — the front end that the user sees is just a thin veneer written in HTML, while the underlying business logic and data storage happens on servers behind the scenes.  The client has to be quite a bit fatter than a Unix “dumb terminal”, but if it can run a web browser, it’s fat enough. 

This was a great idea in many ways.  Multi-tiered development encourages encapsulation and data locality.  Encapsulating the back end means that you can write multiple front-end clients, and shipping the clients around as HTML means that you can automatically update every client to the latest version of the front end.  My job from 1996 to 2001 was to work on the implementation of what became the primary languages used on the front-end tier (JScript) and the web server tier (VBScript).  It was exciting work.

Right now, we’re looking to the future.  We’ve made a good start at letting people develop thin-client multi-tiered applications in Windows, but there is a lot more we can do.  To do so, we need to understand what exactly is goodness.  So let me declare right now Eric Lippert’s Rich Client Manifest

The thin-client multi-tiered approach to software development squanders the richness available on the vast majority of client platforms that I’m interested in.  We must implement tools that allow rich client application developers to attain the benefits of the thin-client multi-tiered model.

That’s the whole point of the .NET runtime and the coming Longhorn API.  The thin client model lets you easily update the client and keeps the business logic on the back tier?  Great — let’s do the same thing in the rich client world, so that developers who want to develop front ends that are more than thin HTML-and-script shells can do so without losing the advantages that HTML-and-script afford. 

Practice

I’ve been thinking about this highfalutin theoretical stuff recently because of some eminently practical concerns.  Many times over the years I’ve had to help out third party developers who have gotten themselves into the worst of both worlds.  A surprisingly large number of people look at the benefits of the thin client model — easy updates (the web), a declarative UI language (HTML), an easy-to-learn and powerful language (JScript) — and decide that this is the ideal environment to develop a rich client application.

That’s a bad idea on so many levels.  Remember, it is called the thin client model for a reason.  I’ve seen people who tried to develop database management systemsin JScript and HTML!  That’s a thorough abuse of the thin client model — in the thin client model, the database logic is done on the backend by a dedicated server that can handle it, written by database professionals in hand-tuned C.  JScript was designed for simple scripts on simple web pages, not large-scale software.

Suppose you were going to design a language for thin client development and a language for rich client development.  What kinds of features would you want to have in each?

For the thin client, you’d want a language that had a very simple, straightforward, learn-as-you-go syntax.  The concept count, the number of concepts you need to understand before you start programming, should be low.  The “hello world” program should be something like

print "hello, world!"

and not

import library System.Output;
public startup class MainClass
{
  public static startup function Main () : void
  {
     System.Output("hello, world!");
  }
};

It should allow novice developers to easily use concepts like variables and functions and loops.  It should have a loose type system that coerces variables to the right types as necessary.  It should be garbage collected.  There must not be a separate compile-and-link step.  The language should support late binding.  The language will typically be used for user interface programming, so it should support event driven programming.  High performance is unimportant — as long as the page doesn’t appear to hang, its fast enough.  It should be very easy to put stuff in global state and access it from all over the program — since the program will likely be small, the lack of locality is relatively unimportant. 

In short, the language should enable rapid development of simple software by relatively unsophisticated programmers through a flexible and dynamic programming model. 

OK, what about the rich-client language?  The language requirements of large-scale software are completely different.  The language must have a rigid type system that catches as many problems as possible before the code is checked in.  There must be a compilation step, so that there is some stage at which you can check for warnings.  It must support modularization, encapsulation, information hiding, abstraction and re-use, so that large teams can work on various interacting components without partying on each other’s implementation details.  The state of the program may involve manipulating scarce and expensive resources — large amounts of memory, kernel objects such as file handles, etc.  Thus the language should allow for fine-grained control over the lifetime of every byte.

Object Oriented Programming in C++ is one language and style that fits this bill, but the concept count of C++ OOP is enormous — pure, virtual, abstract, instance, static, base, pointers, references…  That means that you need sophisticated, highly educated developers.  The processing tasks may be considerable, which means that performance becomes a factor.  Having a complex “hello world” is irrelevant, because no one uses languages like this to write simple programs.

In short, a rich-client language should support large-scale development of complex software by large teams of sophisticated professional programmers through a rigid and statically analyzable programming model.

Complete opposites!  Now, what happens when you try to write a rich client style application using the thin client model? 

Apparent progress will be extremely rapid — we designed JScript for rapid development.  Unfortunately, this rapid development masks serious problemsfestering beneath the surface of apparently working code, problems which will not become apparent until the code is an unperformant mass of bugs. 

Rich client languages like C# force you into a discipline — the standard way to develop in C# is to declare a bunch of classes, decide what their public interfaces shall be, describe their interactions, and implement the private, encapsulated, abstracted implementation details.  That discipline is required if you want your large-scale software to not devolve into an undebuggable mess of global state.  If you can modularize a program, you can design, implement and test it in independent parts.

It is possible to do that in JScript, but the language does not by its very nature lead you to do so.  Rather, it leads you to to favour expedient solutions (call eval!) over well-engineered solutions (use an optimized lookup table).  Everything about JScript was designed to be as dynamic as possible.

Performance is particularly thorny.  Traditional rich-client languages are designed for speed and rigidity.  JScript was designed for comfort and flexibility.  JScript is not fast, and it uses a lot of memory.  Its garbage collector is optimized for hundreds, maybe thousands of outstanding items, not hundreds of thousands or millions.

So what do you do if you’re in the unfortunate position of having a rich client application written in a thin-client language, and you’re running into these issues?

It’s not a good position to be in.

Fixing performance problems after the fact is extremely difficult.  The way to write performant software is to first decide what your performance goals are, and then to MEASURE, MEASURE, MEASURE all the time.  Performance problems on bloated thin clients are usually a result of what I call “frog boiling”.  You throw a frog into a pot of boiling water, it jumps out.  You throw a frog into a pot of cold water and heat it up slowly, and you get frog soup.  That’s what happens — it starts off fine when it is a fifty line prototype, and every day it gets slower and slower and slower… if you don’t measure it every day, you don’t know how bad its getting until it is too late.  The best way to fix performance problems is to never have them in the first place.

Assuming that you’re stuck with it now and you want to make it more usable, what can you do?

• Data is bad. Manipulate as little data as possible.  That’s what the data tier is for.  If you must manipulate data, keep it simple — use the most basic data structures you can come up with that do the job.
• Code is worse.  Every time you call eval, performance sucks a little bit more.  Use lookup tables instead of calling eval.  Move code onto the server tier. 
• Avoid closures.  Don’t nest your functions unless you really understand closure semantics and need them.
• Do not rely on “tips and tricks” for performance.  People will tell you “declared variables are faster than undeclared variables” and “modulus is slower than bit shift” and all kinds of nonsense.  Ignore them.  That’s like mowing your lawn by going after random blades of grass with nail scissors.  You need to find the WORST thing, and fix it first.  That means measuring.  Get some tools — Visual Studio Analyzer can do some limited script profiling, as can the Numega script profiler, but even just putting some logging into the code that dumps out millisecond timings is a good way to start.  Once you know what the slowest thing is, you can concentrate on modularizing and fixing it.
• Modularize.  Refactor the code into clean modules with a well-defined interface contract, and test modules independently. 

But the best advice I can give you is simply use the right tool for the right job.  The script languages are fabulous tools for their intended purpose.  So are C# and C++.  But they really are quite awful at doing each other’s jobs!

You might have noticed something odd about that last example using SetValue. The CLR documentation it notes that the function signature is:

public function SetValue(value : Object, indices : int[]) : void

The indices parameter is typed as taking a .NET array of integers but in the example in my last entry we give it a literal JScript array, not a CLR array.

JScript .NET arrays and CLR arrays work together, but because these two kinds of arrays are so different they do not work together perfectly. The problem is essentially that JScript .NET arrays are much more dynamic than CLR arrays. JScript .NET arrays can change size, can have elements of any type, and so on.

The rules for when JScript .NET arrays and CLR arrays may be used in place of each other are not particularly complicated but still you should exercise caution when doing so. In particular, when you use a JScript .NET array in a context where a CLR array is expected you can get unexpected results. Consider this example:

function ChangeArray(arr : int[]) : void
{
  print(arr[0]); // 10
  arr[0] += 100;
}
var jsarr : Array = new Array(10, 20, 30);
ChangeArray(jsarr);
print(jsarr[0]); // 10 or 110?

This might look like it prints out 10 then 110, but in fact it prints out 10 twice. The compiler is unable to turn the dynamic JScript array into a reference to a CLR array of integers so it does the next best thing. It makes a copy of the JScript array and passes the copy to the function. If the function reads the array then it gets the correct values. If it writes it, then only the copy is updated, not the original.

To warn you about this possibly unintentional consequence of mixing array flavours, the compiler issues the following warning if you do that:

warning JS1215: Converting a JScript Array to a System.Array 
results in a memory allocation and an array copy

You may now be wondering then why the call to SetValue which had the literal JScript .NET array did not prompt this warning. The warning is suppressed for literal arrays. In the case of literal arrays the compiler can determine that a literal array is being assigned to a variable of CLR array type. The compiler then optimizes away the creation of the JScript .NET array and generates code to create and initialize the CLR array directly. Since there is then no performance impact or unexpected copy, there is no need for a warning.

Note that if every element of the source JScript .NET array cannot be converted to the element type of the CLR array then a type mismatch error will result. For instance, this would fail:

var arr1 : int[] = new Array(10, "hello", 20); // Type mismatch error at runtime
var arr2 : int[] = [10, "hello", 20];          // Type mismatch error at compile time

Note also that this applies to multidimensional arrays. There is no syntax for initializing a multidimensional array in JScript .NET:

var mdarr : int[,] = [ [ 1, 2 ], [3, 4] ]; // Nice try, but illegal

A rectangular multidimensional array is not “an array of arrays”. In this case you are assigning a one-dimensional array which happens to contain arrays to a two-dimensional array of integers; that is not a legal assignment. If, however, you want a ragged array it is perfectly legal to do this:

var ragged : int[][] = [ [ 1, 2 ], [3, 4] ];
print(ragged[1][1]); // 4

Rectangular multidimensional arrays are indexed with a comma-separated list inside one set of square brackets. If you use ragged arrays to simulate true multidimensional arrays then the indices each get their own set of brackets.

Note that JScript .NET arrays do not have any of the methods or properties of a CLR array. (Strings, by contract, can be used implicitly as either JScript .NET strings or as System.String values, which I’ll talk more about later.)  But JScript .NET arrays do not have CLR array fields like Rank, SetValue, and so on.

Next time on FAIC: I’ll talk a bit about going the other way — using a CLR array where a JScript array is expected.

Commentary from 2021:

Unlike the previous episode, this episode solicited some good comments on how BeyondJS / BeyondRhino deal with the very similar problem of making JS arrays interoperate with Java arrays, so that JS scripts could be used to more easily write unit tests for JVM programs. There was a lot of this sort of parallel work going on at the time.

As I was saying the other day, CLR arrays and JScript arrays are totally different beasts. It is hard to imagine two things being so different and yet both called the same thing. Why did the CLR designers and the JScript designers start with the same desire — create an array system — and come up with completely different implementations?

Well, the CLR implementers knew that dense, nonassociative, typed arrays are easy to make fast and efficient. Furthermore, such arrays encourage the programmer to keep homogenous data in strictly bounded tables. That makes large programs that do lots of data manipulation easier to understand. Thus, languages such as C++, C# and Visual Basic have arrays like this, and thus they are the basic built-in array type in the CLR.

Sparse, associative, untyped arrays are not particularly fast but they are far more dynamic and flexible than Visual Basic-style arrays. They make it easy to store heterogeneous data in any table without worrying about picky details such as exactly how big that table is. In other words, they are “scripty”. Languages such as JScript and Perl have arrays like this.

JScript .NET has both very dynamic, scripty arrays and more strict CLR arrays, making it suitable for both rapid development of scripts and programming in the large. But like I said, making these two very different kinds of arrays work well together is not trivial.

JScript .NET supports the creation of multidimensional typed arrays. As with single-dimensional arrays, the array size is not part of the type. To annotate a variable as containing a typed multidimensional array the syntax is to follow the type with brackets containing commas. For example, to annotate a variable as containing a two dimensional array of Strings you would say:

var multiarr : String[,];

The number of commas between the brackets plus one is equal to the rank of the array. (By this definition if there are no commas between the brackets then it is a rank-one array, as we have already seen.)

A multidimensional array is allocated with the new keyword as you might expect:

multiarr = new String[4,5];
multiarr[0,0] = "hello";

Notice that elements of typed arrays are always accessed with a comma-separated list of integer indices. There must always be exactly one index for each dimension in the array. You can’t use the ragged array syntax [0][0].

There are certain situations in which you know that a variable or function argument will refer to a CLR array but you do not actually know the element type or the rank, just that it is an array. Should you find yourself in one of these (rather rare) situations there is a special annotation for a CLR array of unknown type and rank:

var sysarr : System.Array;
sysarr = new String[4,5];
sysarr = new double[10];

As you can see, a variable of type System.Array may hold any CLR array of any type and rank. However, there is a drawback. Variables of type System.Array may not be indexed directly because the rank is not known. This is illegal:

var sysarr : System.Array;
sysarr = new String[4,5];
sysarr[1,2] = "hello";  // ILLEGAL, System.Arrays are not indexable

Rather, to index a System.Array you must call the GetValue and SetValue methods with an array of indices:

var sysarr : System.Array;
sysarr = new String[4,5];
sysarr.SetValue("hello", [1,2]);

The rank and size of a System.Array can be determined with the Rank, GetLowerBound and GetUpperBound members.

Thinking about this a bit now, I suppose that we could have detected at compile time that a System.Array was being indexed, and constructed the call to the getter/setter appropriately for you, behind the scenes.  But apparently we didn’t.  Oh well.

Next time on FAIC: mixing and matching JScript and CLR arrays.

Commentary from 2021:

As I say rather a lot, design is the art of coming up with compromises, and of course a good compromise leaves everybody mad. I expected more pushback on these decisions; trying to make it all work in a manner that felt both true to JScript and made efficient use of CLR types designed for a statically-typed world led to a lot of compromises to be mad about. But it went over pretty well, insofar as JScript .NET went over anywhere at all.

There was some good discussion in the comments on the original post about how you might write a fold so that it worked on JScript arrays, CLR arrays, multidimensional arrays, and so on, and did so efficiently.

One of the major differences between JScript .NET and JScript Classic is that JScript .NET now supports optional type annotations on variables.  The number of built-in primitive types has also increased dramatically.  JScript .NET adds value types boolean, byte, char, decimal, double, float, int, long, sbyte, short, uint, ulong and ushort.  In addition, JScript .NET integrates its type system with the CLR type system — a string in JScript has all the properties and methods of the string prototype and all the properties and methods of a System.String.  Backwards compatibility and interoperability with the CLR were two very important design criteria.

The primitive types are pretty straightforward though.  Some more interesting stuff happens when we think about how complex types like arrays interoperate between JScript .NET and the CLR. Let’s quickly review the terminology:

A sparse array may have “holes” in the valid indices. A sparse array with three elements might have elements 0, 10 and 1000 defined but nothing in between. The opposite of a sparse array is a dense array. In a dense array all the indices between the lowest and highest indices are valid indices. A dense array with elements 0 and 1000 has 1001 elements.

A fixed-size> array has a particular valid range of indices. Typically the size is fixed when the array is created and it may not be changed. A variable-sized array does not have any maximum size. Elements may be added or removed at any time.

A single-dimensional array maps a single index onto a value. A multi-dimensional array may require any number of indices to fetch a value.

The number of dimensions an array has is called its rank. (Other terms
such as dimensionality or arity are occasionally used but we will stick to rank.)

A uniformly-typed array has every element of the same type. An any-typed array may have elements of any type.

An associative array is an array where the indices are strings. A nonassociative array has integer indices.

A literal array is a JScript .NET array defined in the actual source code, much as “abcde” is a literal string or 123.4 is a literal number. In JScript .NET a literal array is a comma-separated list of items inside square brackets:

var arr = [10, 20, "hello"];
var item = arr[1]; // 20

JScript arrays are sparse, variable-sized, single-dimensional, any-typed, associative arrays. CLR arrays are the opposite in every way! They are dense, fixed-size, multi-dimensional, uniformly-typed, nonassociative arrays. It is hard to imagine two more different data structures with the same name.  Making them interoperate at all was a pain in the rear, believe me.

This is a pretty big topic, so I think I’ll split it up over a few entries.  Let me talk a bit about annotation and typing, and we’ll pick up where we left off tomorrow.

Traditional JScript arrays are soft-typed; they can store heterogeneous data:

var arr = new Array();
arr[0] = "hello";
arr[1] = 123.456;
arr[2] = new Date();

CLR arrays, on the other hand, are uniformly typed. Every element of a CLR array is the same type as every other element. This difference motivates the type annotation syntaxes for each. In JScript .NET the traditional arrays are annotated with the Array type and CLR arrays are annotated with the type of the element followed by []:

var jsarr : Array = new Array()
jsarr[0] = "hello";
var sysarr : double[] = new double[10];
sysarr[0] = 123.4;

Note that CLR arrays are fixed-size, but the size is not part of the type annotation; sysarr can be a one-dimensional array of double of any size. This is perfectly legal, for example:

var sysarr : double[] = new double[10];
sysarr[0] = 123.4
sysarr = new double[5];

This throws away the old array and replaces it with a new, smaller array. But once a typed array is created it may not be resized.

Tomorrow: true multidimensional arrays in JScript .NET.

—-

Commentary from 2021:

This post was one of the very few over the lifetime of my blog that got no comments. As an introductory post to a rather arcane subject, I suppose that is not particularly surprising. I put a lot of work into arrays in JS.NET and it was not very well documented, so I figured that getting it out in my blog would be generally helpful.

A number of readers made some good comments on my article on JScript typing that deserve to be called out in more detail.

First, I was being a little sloppy in my terminology — I casually conflated static typing with strong typing, and dynamic typing with weak typing. Thanks for calling me on that. Under the definitions proposed by the reader, JScript would be a dynamically typed language (because every variable can take a value of any type) and a strongly typed language (because every object knows what type it is.) By contrast, C++ is a statically typed language (because every variable must have a type, which the compiler enforces) but also a weakly typed language (because the reinterpret cast allows one to turn pointers into integers, and so on.)

Second, a reader notes that one of the shortcomings of JScript is that though it is a strongly typed language (in our new sense) that it is a royal pain to actually determine the runtime type an object. The typeof operator has a number of problems:

• null is listed as being of the object type, though technically it is a member of the Null type.
• primitives (strings, numbers, Booleans) wrapped in objects are listed as being of the object type rather than their underlying type.
• JScript, unlike VBScript, does not interrogate COM objects to determine the class name.
• If JScript is passed a variant from the outside world that it cannot make sense of then typeof returns “unknown”.

Perhaps there is some other way. Prototype inheritance affords a kind of type checking, for example.

Prototype inheritance works like this: every JScript object has an object (or possibly null) called its prototype object.  Suppose an object foo has prototype object bar, and bar has prototype object baz, and baz has prototype object null. If you call a method on foo then JScript will search foo, bar and baz for that method, and call the first one it finds.

The idea is that one object is a prototypical object, and then other objects specialize it. This allows for code re-use without losing the ability to dynamically customize behaviour of individual objects.

Prototypes are usually done something like this:

var Animal = new Object();
// omitted: set up Animal object
function Giraffe()
{
  // omitted: initialize giraffe object.
}
Giraffe.prototype = Animal;
var Jerry = new Giraffe();

Now Jerry has all the properties and methods of an individual Giraffe object AND all the properties and methods of Animal.  You can use IsPrototypeOf to see if a given object has Animal on its prototype chain. Since prototype chains are immutable once created, this gives you a pretty reliable sort of type checking.

Note that Giraffe is not a prototype of Jerry. Note also that Animal is not the prototype of Giraffe! The object which is assigned to the prototype property of the constructor is the prototype of the instance.

Now, you all are not the first people to point out to me that determining types is tricky. A few years ago someone asked me what the differences are amongst

if (func.prototype.IsPrototypeOf(instance))

and

if (instance.constructor == func)

and

if (instance instanceof func)

The obvious difference is that the first one looks at the whole prototype chain, whereas the second two look at the constructor, right? Or is that true? Is there a semantic difference between the last two?

There is. Let’s look at some examples, starting with one that seems to show that there is no difference:

function Car() { }
var honda = new Car();
print(honda instanceof Car); // true
print(honda.constructor == Car);  // true

It appears that instance instanceof func and instance.constructor == func have the same semantics. They do not. Here’s a more complicated example that demonstrates the difference:

var Animal = new Object();
function Reptile() { }
Reptile.prototype = Animal;
var lizard = new Reptile();
print(lizard instanceof Reptile); // true
print(lizard.constructor == Reptile); // false

In fact lizard.constructor is equal to Object, not Reptile.

Let me repeat what I said above, because no one understands this the first time — I didn’t, and I’ve found plenty of Javascript books that get it wrong. When we say

Reptile.prototype = Animal;

this does not mean “the prototype of Reptile is Animal“.  It cannot mean that because (obviously!) the prototype of Reptile, a function object, is Function.prototype.  No, this means “the prototype of any instance of Reptile is Animal“.  There is no way to directly manipulate or read the prototype chain of an existing object in JScript.

Now that we’ve got that out of the way, the simple one first:

instance instanceof func means “is the prototype property of func equal to any object on instance’s prototype chain?”  So in our second example, the prototype property of Reptile is Animal and Animal is on lizard‘s prototype chain.

But what about our first example where there was no explicit assignment to the Car prototype?

The compiler creates a function object called Car.  It also creates a default prototype object and assigns it to Car.prototype.  So again, when we way

print(honda instanceof Car);

the instanceof operator gets Car.prototype and compares it to the prototype chain of honda. Since honda was constructed by Car it gets Car.prototype on its prototype chain.

To sum up the story so far,  instance instanceof func is actually a syntactic sugar for func.prototype.IsPrototypeOf(instance) This explains why lizard instanceof Reptile returns true — Reptile.prototype is a prototype of lizard.

So what the heck is going on with the constructor property then?  How is it possible that we can say lizard = new Reptile(); and at the same time lizard.constructor == Reptile is false???

Let’s go back to our simple first example.  I said above that since Car has no prototype assigned to it, we create a default prototype.  During the creation of the default prototype, the interpreter assigns Car to Car.prototype.constructor.  That might be a little confusing, so let’s look at some pseudocode.  This:

function Car(){}

logically does the same thing as

var Car = new Function();
Car.prototype = new Object();
Car.prototype.constructor = Car;

Now we say

var honda = new Car();
print(honda.constructor == Car );

and what happens? honda has no constructor property, so it looks on the prototype chain for any object with a constructor property. In this case Car.prototype is on the prototype chain and it has a constructor property equal to Car, so the comparison is true.  Remember, any property of an object’s prototype object is treated as a property of the object itself – that’s what “prototype” means.

Now let’s look at our second example:

var Animal = new Object();
function Reptile(){ }
Reptile.prototype = Animal;

Logically this does the same thing as

var Animal = new Object();
var Reptile = new Function();
Reptile.prototype = new Object();
Reptile.prototype.constructor = Reptile;
Reptile.prototype = Animal;

Whoops. The default prototype has been thrown away.  Now when we say

print(lizard.constructor == Reptile );

what happens? lizard does not have a constructor property, so we look at the prototype chain and find Animal.  But Animal also does not have a constructor property either! So we look on Animal‘s prototype chain. Animal was constructed via new Object() so therefore it has Object.prototype on its prototype chain, and Object.prototype has a constructor property.  As you might expect from our previous discussion of how the constructor property is initialized, Object.prototype.constructor is set to Object.

Therefore lizard.constructor is equal to Object, not Reptile, even though lizard is an instance of Reptile and was constructed by the Reptile function object!

You would think that the script engine would automatically assign the constructor property to the object when it was constructed, but it does not. It assigns the property to the prototype and relies on prototype inheritance. I was not a member of the ECMAScript committee when this decision was made, so I don’t know why we standardized this rather bizarre behaviour, but we’re stuck with it now!

Notes from 2020:
This article is one I referred back to so many times over the years I worked on JScript; the prototype mechanism can be very confusing.

There were a few interesting comments when I first wrote this. A couple highlights:

• In some implementations of ECMAScript you can manipulate the proto chain directly with the __proto__ property. Make sure you understand what you’re doing before you change it!
• Using “new String” to make a string can lead to an object that has different properties than just making the “primitive” string. This is by design — bad design, but design nevertheless. This was the subject of part three of this series.

I’m going to make a rare departure from technical stuff for a moment.  I’ve seen Matrix Revolutions twice in the last twelve hours, once in IMAX.

A number of people have asked me for an opinion.  To sum up in a spoiler-free manner:

First off, IMAX was better, though not enormously so.  (Oddly enough, Reloaded was enormously better in IMAX.)  As an example of state-of-the-art action movie making, it kicked ass.  But what about the underlying theme?  Let’s summarize:

The Matrix was a movie about kung fu vs. evil robots that asked and answered the philosophical question “what is the nature of reality?”

The Matrix Reloaded was a movie about kung fu vs. evil robots that asked and answered the philosophical question “what is the nature of free will?”

Matrix Revolutions was a movie about kung fu vs. evil robots that asked and answered the philosophical question “if Neo got into a fight with Agent Smith, who would win?”

We return you now to your regularly scheduled programming.

Commentary from 2019:

As I’ve noted before, the VSTO team code names and whatnot were all taken from The Matrix, and it was a fun theme.

I am not “spoiler averse” but I saw the first movie in the series knowing absolutely nothing about it, and was thrilled to discover that it had kung fu, evil robots, and literal brain-in-a-vat philosophy undergrad thought experiments. Sure, lots of it was hokey and unbelievable, but it had such energy, and was genuinely surprising.

I should not have been surprised by a reversion to the mean in the sequels; the original was hard to top.

I thought I might spend a few days talking about the JScript and JScript .NET type systems, starting with some introductory material.

Consider a JScript variable:

var myVar;

Now think about the possible values you could store in the variable. A variable may contain any number, any string or any object. It can also be true or false or null or even undefined. This is a rather large set of possible values. In fact, the set of all legal values is infinite. Countably infinite, and in practice limited by available memory, but in theory there is no upper limit.

A type is characterized by two things, a set and a rule. First, a type consists of a subset (possibly infinitely large) of the set of all possible values. Second, a type defines a rule for transforming values outside the set into values in the set. (This rule may specify that certain values are not convertible and hence produce “type mismatch” errors.)

For example, String is a type. The set of all possible strings is an (infinite) subset of the set of all possible values, and there are rules for determining how all non-string values are converted into strings.

JScript Classic is a dynamically typed language. This means that any value of any type may be assigned to any variable without restriction. It is often said — inaccurately — that “JScript has only one type”. This is true only in the sense that JScript has no restrictions on what data may be assigned to any variable, and in that sense every variable is “the same type” – namely, the “any possible value” type. However, the statement is misleading because it implies that JScript supports no types at all, when in fact it supports six built-in types.

JScript .NET, by contrast, is an optionally statically-typed language. A JScript .NET variable may be given a type annotation which restricts the values which may be stored in the variable. This annotation is optional; an unannotated variable acts like a JScript variable and may be assigned any value.

JScript has the property that a value can always describe its own type at runtime. This is not true in, say, C, where you can have a void* and no way of asking it “are you pointing to an integer or a string variable?” In JScript, you can always ask a value what its type is and it will tell you.

The concept of subtyping is not particularly important in JScript Classic though it will become quite useful when we discuss JScript .NET classes later. Essentially a type T1 is a subtype of another type T2 if T1’s set of values is a subset of T2’s set of values. A type consisting of the set of all integers might be a subtype of a type consisting of all the numbers, for instance. (This is not how the integers are traditionally construed; the C type system makes integers and floats disjoint types, where the integer 1 and the float 1.0 are different values that happen to compare as equal — but comparisons across types is a subject for a later blog entry.)

Anyway, JScript Classic has six built-in types, all of which are disjoint. They are as follows:

• The Number type contains all floating-point numbers as well as positive Infinity, negative Infinity and a special Not-a-Number (“NaN”) values. It may seem odd that “Not-a-Number” is a Number but this does in fact make sense. NaN is the value returned when an operation logically must return a number but no actual number makes sense. For example, when trying to convert the string "banana" to a Number, NaN is the result. Because numbers in JScript are actually represented by a 64 bit floating point number there are a finite number of possible Number values. The number of numbers is very large (in fact there are 18437736874454810627 possible numbers, which is just shy of 2^64.) Numbers have approximately fifteen decimal digits of precision and can range from as tiny as 2.2 x 10^-308 to as large as 1.7 x 10³⁰⁸.
• The String type contains all Unicode strings of any length (including zero-length empty strings.) The string type is for all practical purposes infinite, as the length of a string is limited only by the ability of the operating system to allocate enough memory to hold it.
• The Boolean type has two values: true and false
• The Null type has one value: null
• The Undefined type has one value: undefined. All uninitialized JScript variables are automatically set to undefined
• The Object type has an infinite number of values. An object is essentially a collection of named properties where each property can be a value of any type. In JScript many things are objects: functions, dates, arrays and regular expressions are all objects.

Types are not themselves “first class” objects in JScript, though they are in JScript .NET. I’ll discuss that, along with the differences between prototype and class inheritance, in later entries.

Notes from 2020

It would be interesting to revisit this series in the context of the TypeScript type system, which is a great example of how rich and powerful a gradually typed language can be. The JScript.NET gradual type system was very simple compared to the TS system!

This series got a lot of good comments from readers. A few highlights from this first episode:

• the typeof operator in JScript bizarrely does not follow the rules of the type system that I just laid out. It identifies null as an object, but does not identify functions as objects, for example. External functions, like DOM functions, may not be identified as functions. It can also return “unknown” in some rare cases.
• Also confusing: typeof(3) is number, but typeof(new Number(3)) is object.
• And also confusing, typeof(this) when called from the body of a user-defined function on the prototype of Number is object.
• A number of inconsistencies and confusions regarding prototypes were also raised; I discuss these in a later episode.
• In the earliest published version of this article I used the terms “strong” and “weak” with respect to the type system, and readers rightly took me to task for that. This was the beginning of a realization that I have since strongly expressed: “strong” and “weak” are so vague that they are meaningless. I haven’t checked lately, but one time when I read the Wikipedia article on strong typing, it listed eleven different contradictory meanings. I’ve since expressed the (strong!) opinion that “strongly typed” simply means “a type system that I admire”. Instead of characterizing type systems as “strong” or “weak”, instead say what properties they really have: what restrictions do they impose, when are they imposed, and in what ways can those restrictions be violated, and what are the consequences?
• Readers similarly noted that “untyped” is vague; it is often used to mean “no restriction is placed on the possible values of variables” (as is the case in classic JavaScript) but is also used to mean “no type system is imposed at all on any values” (as is the case in, say, untyped lambda calculus, where all values are function from function to function; there are no integers or strings at all.)

As I promised, more information on why eval is evil. (We once considered having T-shirts printed up that said “Eval is evil!” on one side and “Script happens!” on the other, but the PM’s never managed to tear themselves away from their web browsing long enough to order them.)

Incidentally, a buddy of mine who is one of those senior web developer kinda guys back in Waterloo sent me an email yesterday saying “Hello, my name is Robert and I am an evalaholic”. People, it wasn’t my intention to start a twelve step program, but hey, whatever works!

As I discussed the other day, eval on the client is evil because it leads to sloppy, hard-to-debug-and-maintain programs that consume huge amounts of memory and run unnecessarily slowly even when performing simple tasks. But like I said in my performance rant, if it’s good enough, then hey, it’s good enough.  Maybe you don’t need to write maintainable, efficient code. Seriously! Script is often used to write programs that are used a couple of times and then thrown away, so who cares if they’re slow and inelegant?

But eval on the server is an entirely different beast. First off, server scenarios are generally a lot more performance sensitive than client scenarios.  On a client, once your code runs faster than a human being can notice the lag, there’s usually not much point in making it faster.  But  as I mentioned earlier, ASP goes to a lot of work to ensure that for a given page, the compiler only runs once. An eval defeats this optimization by making the compiler run every time the page runs! On a server, going from 25 ms to 40 ms to serve a page means going from 40 pages a second to 25 pages a second, and that can be expensive in real dollar terms.

But that’s not the most important reason to eschew eval on the server.  Any use of eval (or its VBScript cousins Eval, Execute and ExecuteGlobal) is a potentially enormous security hole:

<%
  var Processor_ProductList;
  var Software_ProductList;
  var HardDisk_ProductList;
  // ...
  CategoryName = Request.QueryString("category");
  ProductList = eval(CategoryName & "_ProductList");
  // ...

What’s wrong with this picture?  The server assumes that the client is not hostile.  Is that a warranted assumption?  Probably not!  You know nothing about the client that sent the request.  Maybe your client page only sends strings like “Processor” and “HardDisk” to the server, but anyone can write their own web page that sends

((new ActiveXObject('Scripting.FileSystemObject')).
DeleteFile('C:*.*',true)); 
Processor

which will cause eval to evaluate

((new ActiveXObject('Scripting.FileSystemObject')).
DeleteFile('C:*.*',true)); 
Processor_ProductList

Obviously that’s a pretty unsophisticated attack.  The attacker can put any code in there that they want, and it will run in the context of the server process.  Hopefully the server process is not a highly privileged one, but still, there’s vast potential for massive harm here just by screwing up the logic on your server.

Never trust the input to a server, and try to never use eval on a server.  Eval injection makes SQL injection look tame!

To try and mitigate these sorts of problems, JScript .NET has some restrictions on its implementation of eval, but that’s a topic for another entry.

Notes from 2020

The attack I’m briefly describing here is of course only one of a great many “hostile client gets bad string onto the server” attack patterns. During my time at Coverity I got to take a deep look at the tools which attempt to detect code paths where a string goes from an untrusted source to a dangerous sink. In terms of the complexity of the analyzed control flows, these were probably the most sophisticated checkers we had, and among the most prone to false positives.

Defeating injection attacks is a hard problem, and I wish we at Microsoft had solved it in the type system, rather than creating a market for expensive third-party solutions that use symbolic execution to effectively do the work of imposing a type system post hoc on an existing program.

Once upon a time I was in high school. Ah, the halcyon days of my youth. One day I was sitting in class, minding my own business when the teacher said: “Does anyone have a thin metal ruler?”

No answer. Apparently no one had a thin metal ruler.

“No? How about a nail file?”

No answer. Now, I cannot imagine that of all the girls in the class, not one of them had a nail file. But I can well imagine that none of them wanted to share it with a teacher.

“No? Hmm.”

So I piped up: “What do you need a nail file for?”

“I have this big staple in this document that I need to remove.”

Upon which point one of my classmates mentioned that he had a staple remover. Problem solved.

Over and over again I find that script customers (both internal consumers here at Microsoft and third-party developers) frequently ask questions like my teacher. That is, they have a preconceived notion of how the problem is going to be solved, and then ask the necessary questions to implement their preconceived solution. And in many cases this is a pretty good technique! Had someone actually brought a thin metal ruler to class, the problem would have been solved. But by choosing a question that emphasizes the solution over the problem, the questioner loses all ability to leverage the full knowledge of the questionees.

When someone asks me a question about the script technologies I almost always turn right around and ask them why they want to know. I might be able to point them at some tool that better solves their problem. And I might also learn something about what problems people are trying to solve with my tools.

Joel Spolsky once said that people don’t want drills, they want holes. As a drill provider, I’m fascinated to learn what kinds of holes people want to put in what kinds of materials, so to speak. Sometimes people think they want a drill when in fact they want a rotary cutter.

Commentary from 2019:

First off, I misattributed that quotation. “People don’t want to buy a quarter-inch drill, they want a quarter-inch hole.” is a quote from the economist Theodore Levitt. At the time I wrote this, I was sure that I had read about this idea in a Joel On Software article, but if I did, I cannot find it now. Apologies for the error.

Second, I did not know at the time that we have a name for this pattern of “have a problem, get a crazy idea about a solution, ask baffling questions about the crazy idea, rather than stating the problem directly” that we see so often on StackOverflow. It is an “XY problem“, which strikes me as a terrible name.

Third, I am reminded of a story about the time I was helping Morton Twillingate put a roof on his shed. “Hand me the screw driver there b’y,” he said so I handed him a Philips head screwdriver. “Sweet t’underin’ Jaysus b’y, give me the screw driver!” he said, pointing at the hammer in my other hand, “If I’d wanted the screw remover I’d have said so!”

The eval method — which takes a string containing JScript code, compiles it and runs it — is probably the most powerful and most misused method in JScript. There are a few scenarios in which eval is invaluable. For example, when you are building up complex mathematical expressions based on user input, or when you are serializing object state to a string so that it can be stored or transmitted, and reconstituted later.

However, these worthy scenarios make up a tiny percentage of the actual usage of eval. In the majority of cases, eval is used like a sledgehammer swatting a fly — it gets the job done, but with too much power. It’s slow, it’s unwieldy, and tends to magnify the damage when you make a mistake. Please spread the word far and wide: if you are considering using eval then there is probably a better way. Think hard before you use eval.

Let me give you an example of a typical usage.

<span id="myspan1"></span>
<span id="myspan2"></span>
<span id="myspan3"></span>
function setspan(num, text)
{
  eval("myspan" + num + ".innerText = '" + text + "'");
}

Somehow the program is getting its hands on a number, and it wants to map that to a particular span. What’s wrong with this picture?

Well, pretty much everything. This is a horrid way to implement these simple semantics. First off, what if the text contains an apostrophe? Then we’ll generate

myspan1.innerText = 'it ain't what you do, it's the way thacha do it';

Which isn’t legal JScript. Similarly, what if it contains stuff interpretable as escape sequences? OK, let’s fix that up.

eval("myspan" + num).innerText = text;

If you have to use eval, eval as little of the expression as possible, and only do it once. I’ve seen code like this in real live web sites:

if (eval(foo) != null && eval(foo).blah == 123)
  eval(foo).baz = "hello";

Yikes! That calls the compiler three times to compile up the same code! People, eval starts a compiler. Before you use it, ask yourself whether there is a better way to solve this problem than starting up a compiler!

Anyway, our modified solution is much better but still awful. What if num is out of range? What if it isn’t even a number? We could put in checks, but why bother? We need to take a step back here and ask what problem we are trying to solve.

We have a number. We would like to map that number onto an object. How would you solve this problem if you didn’t have eval? This is not a difficult programming problem! Obviously an array is a far better solution:

var spans = new Array(null, myspan1, myspan2, myspan3);
function setspan(num, text)
{
  if (spans[num] != null)
    spans[num].innertext = text;
}

Since JScript has string-indexed associative arrays, this generalizes to far more than just numeric scenarios. Build any map you want. JScript even provides a convenient syntax for maps!

var spans = { 1 : mySpan1, 2 : mySpan2, 12 : mySpan12 };

Let’s compare these two solutions on a number of axes:

Debugability: what is easier to debug, a program that dynamically generates new code at runtime, or a program with a static body of code? What is easier to debug, a program that uses arrays as arrays, or a program that every time it needs to map a number to an object it compiles up a small new program?

Maintainability: What’s easier to maintain, a table or a program that dynamically spits new code?

Speed: which do you think is faster, a program that dereferences an array, or a program that starts a compiler?

Memory: which uses more memory, a program that dereferences an array, or a program that starts a compiler and compiles a new chunk of code every time you need to access an array?

There is absolutely no reason to use eval to solve problems like mapping strings or numbers onto objects. Doing so dramatically lowers the quality of the code on pretty much every imaginable axis.

It gets even worse when you use eval on the server, but that’s another post.

Notes from 2020

This was my first deliberately-multi-episode topic.

There were many great comments on this article on the original blog site; to summarize a few of them:

• There are a number of scenarios where you want to dynamically create a new function, but “new Function” is the appropriate choice rather than “eval” most of the time.
• However, the scoping rules for “new Function” and “eval” are different — thanks, JavaScript — and so sometimes there are scenarios where you are forced to eval a new function.
• I was not then and am not now an expert on the browser’s object model. I have many times noted the irony that as a developer of the JS compiler, I was an expert on the inner workings of the JS compiler, and not on how it was used in practice. A reader pointed out that none of my solutions were good practice compared with the expediency of:

var span = document.all("myspan" + num);
if (span != null) span.innertext = text;

or, equivalently, getElementById, on browsers which supported it at the time.

• One reader pointed out that good enough is, by definition, good enough. We’re writing scripts here; if they are fast enough, robust enough, debuggable enough, and so on, then who cares if there is a better way? Why take the time to learn the right way to do something when there is a good enough expedient way you already know?  My answer to those (entirely reasonable) questions would be: JS code is no longer trivial little ten-liner scripts to automate UI elements. We have serious engineering problems to solve, and we should take a disciplined, best-practices-engineering stance by default. In short: move your standards of what is good enough! That benefits all the stakeholders.
• I alluded to, but did not call out explicitly, that eval can be used to deserialize JSON. But it is not quite so easy as that! In 2007 The Curious Schemer published a great follow-up to this article pointing out some of the unexpected ways in which eval does not match JSON.
• Also in 2007, my good friend Rob Hahn and I discussed some alternatives to eval for a tricky problem he had in function creation.