Author Archives: ericlippert

Unknown's avatar

About ericlippert

http://ericlippert.com

Why no var on fields?

Posted on by
2

In my recent request for “things that make you go hmmm”, a reader notes that you cannot use “var” on fields. Boy, would I ever like that. I write this code all the time:

private static readonly Dictionary<TokenKind, string> niceNames =
  new Dictionary<TokenKind, string>()
  {
    { TokenKind.Integer, "int" }, 
    ...
  };

Yuck. It would be much nicer to be able to write

private static readonly var niceNames = 
  new Dictionary<TokenKind, string>()...

You’d think this would be straightforward; we could just take the code that we use to determine the type of a local variable declaration and use it on a field. Unfortunately, it is not nearly that easy. Doing so would actually require a deep re-architecture of the compiler.

Let me give you a quick oversimplification of how the C# compiler works. First we run through every source file and do a “top level only” parse. That is, we identify every namespace, class, struct, enum, interface, and delegate type declaration at all levels of nesting. We parse all field declarations, method declarations, and so on. In fact, we parse everything except method bodies; those, we skip and come back to them later.

Once we’ve done that first pass we have enough information to do a full static analysis to determine the type of everything that is not in a method body. We make sure that inheritance hierarchies are acyclic and whatnot. Only once everything is known to be in a consistent, valid state do we then attempt to parse and analyze method bodies. We can then do so with confidence because we know that the type of everything the method might access is well known.

There’s a subtlety there. The field declarations have two parts: the type declaration and the initializer. The type declaration that associates a type with the name of the field is analyzed during the initial top-level analysis so that we know the type of every field before method bodies are analyzed. But the initialization is actually treated as part of the constructor; we pretend that the initializations are lines that come before the first line of the appropriate constructor.

So immediately we have one problem; if we have “var” fields then the type of the field cannot be determined until the expression is analyzed, and that happens after we already need to know the type of the field.

But it gets worse. What if the field initializer in a “var” field refers to another (static) “var” field? What if there are long chains, or even cycles in those references? There can be arbitrary expressions in those initializers, expressions which contain lambdas which contain expressions which require method type inference or overload resolution. All of these algorithms that are in the compiler were written with the assumption that when they run, the types of every top-level program entity is already known. All of those algorithms would have to be rewritten and tested in a world where top-level type information is being determined from them rather than being consumed by them.

It gets worse still. If you have “var” fields then the initializer could be of anonymous type. Suppose the field is public. There is not yet any standard in the CLR or the CLS about what the right way to expose a field of anonymous type is. We don’t have good policies for documenting them, versioning them, or interoperating with them across languages. Doing this feature would potentially cause huge costs across the division.

Inferred locals have none of these problems; inferred locals never have cycles or refer to things that haven’t been analyzed yet. Inferred locals never escape into public visibility.

So apparently this simple-seeming feature has the potential to cause really, really bad implementation issues in multiple ways, and all in order to avoid a small redundancy. This seems like it is possibly not worth the cost. If our goal is to remove the redundancy, I would therefore prefer to remove it the other way. Make this legal:

private static readonly Dictionary<TokenKind, string> niceNames =
  new()...

That is, state the type unambiguously in the declaration and then have the “new” operator be smart about figuring out what type it is constructing based on what type it is being assigned to. This would be much the same as how the lambda operator is smart about figuring out what its body means based on what it is being assigned to.

Reflections from 2020

  • This was written in 2009; the compiler of course was rearchitected by the Roslyn project, and it is now much more “demand driven” in terms of what gets analyzed when, so some of the objections made in this article no longer apply. However, many of them still do. Changing how type inference works at the “global” level has large design, implementation and testing costs, and that is time, money and effort that gets taken away from more important features than saving a few keystrokes.
  • The syntax I proposed at the end may be incorporated into C# 9; this idea has been coming up in the design team discussions for a long time now!

 

 

Covariance and Contravariance, Part 11: To infinity, but not beyond

Posted on by
4

As I’ve discussed at length in this space, we are considering adding covariance and contravariance on delegate and interface types parameterized with reference types to a hypothetical future version of C#. (See my earlier articles in this series for an explanation of the proposed feature.)

Variance is highly useful in mainstream cases, but exposes a really irksome problem in certain bizarre corner cases. Here’s just one example.

Consider a “normal” interface contravariant in its sole generic type parameter, and a “crazy” invariant interface which extends the normal interface in a strange way:

public interface IN<in U> {}
public interface IC<X> : IN<IN<IC<IC<X>>>> {}

This is a bit weird, but certainly legal.

Before I go on, I want to digress and talk a bit about why this is legal. Most people when they first see such a beast immediately say “but an interface cannot inherit from itself, that’s an illegal circularity in the inheritance chain!”

First off, no, that is not correct. Nowhere does the C# specification make this kind of inheritance illegal, and in fact, a weak form of it must be legal. You want to be able to say:

interface INumber<X> : IComparable<INumber<X>> { ... }

that is, you want to be able to express that one of the guarantees of the INumber<X> contract is that you can always compare one number with another one. Therefore, it must be legal to use a type’s name in a type argument of a generic base type.

However, all is not rosy. This particularly gross kind of inheritance that I give as an example is in fact illegal in the CLR, even though it is not illegal in C#. This means that it is possible to have the C# compiler generate an interface type which then cannot be loaded by the CLR. This unfortunate mismatch is troubling, and I hope in a future version of C# to make the type definition rules of C# as strict or stricter than those of the CLR. Until then, if it hurts when you do that, don’t do it.

Update from 2020: I never did. I left Microsoft in 2012 and I do not know if anyone else ever did either.

Second, unfortunately, the C# compiler presently has numerous bugs in its cycle detector such that sometimes things which kinda look like cycles but are in fact not cycles are flagged as cycle errors. This just makes it all the more difficult for people to understand what is a legal cycle and what isn’t. For example, the compiler today will incorrectly report that this is an illegal base class cycle, even though it clearly is not:

public class November<T> {}
public class Romeo : November<Romeo.Sierra.Tango> {
   public class Sierra {
       public class Tango {}
    }
}

I have devised a new (and I believe correct!) cycle detection algorithm implementation, but unfortunately it will not make it into the service release of the C# 3 compiler. It will have to wait for a hypothetical future release. I hope to address the problem of bringing the legal type checker into line with the CLR at the same time.

Update from 2020: The implementation I wrote for the C# 3 service release was judged too large and risky a change. I seem to recall that I eventually added it to C# 4 but I am hazy on the details. Certainly none of these problems are in Roslyn, which has a completely different algorithm for analyzing base types.

Anyway, back to the subject at hand: crazy variance. We have the interfaces defined as above, and then give the compiler a little puzzle to solve:

IC<double> bar = whatever;
IN<IC<string>> foo = bar;  // Is this assignment legal?

I am about to get into a morass of impossible-to-read generic names, so to make it easier on all of us, I am going to from now on abbreviate IN<IC<string>> as NCS. IC<double> will be abbreviated as CD. You get the idea I’m sure.

Similarly, I will notate “is convertible to by implicit reference conversion” by a right-pointing arrow. So the question at hand is true or false: CD→NCS ?

Well, let’s see. Clearly CD does not go to NCS directly. But (the compiler reasons) maybe CD’s base type does.

CD’s base type is NNCCD. Does NNCCD→NCS? Well, N is contravariant in its parameter so therefore this boils down to the question, does CS→NCCD ?

Clearly not directly. But perhaps CS has a base type which goes to NCCD. The base type of CS is NNCCS. So now we have the question does NNCCS→NCCD ?

Well, N is contravariant in its parameter, so this boils down to the question does CCD→NCCS ?

Let’s pause and reflect a moment here.

The compiler has “reduced” the problem of determining the truth of CD→NCS to the problem of determining the truth of CCD→NCCS! If we keep on “reducing” like this then we’ll get to CCCD→NCCCS, CCCCD→NCCCCS, and so on.

I have a prototype C# compiler which implements variance – if you try this, it says “fatal error, an expression is too complex to compile”.

I considered implementing an algorithm that is smarter about determining convertibility; the paper I reference below has such an algorithm. (Fortunately, the C# type system is weak enough that determining convertibility of complex types is NOT equivalent to the halting problem; we can find these bogus situations both in principle and in practice. Interestingly, there are type systems in which this problem is equivalent to the halting problem, and type systems for which the computability of convertibility is still an open question.) However, given that we have many other higher priorities, it’s easier to just let the compiler run out of stack space and have a fatal error. These are not realistic scenarios from which we must sensibly recover.

This is just a taste of some of the ways that the type system gets weird. To get a far more in-depth treatment of this subject, you should read this excellent Microsoft Research paper.

Update from 2020:

Though I implemented a prototype of Andrew Kennedy’s circular reference detector, I did not ever get it into any version of C#. We just kept saying “expression too complex” when the compiler ran out of stack. I seem to recall that it was eventually added to Roslyn but I haven’t checked the sources to be sure.

The question that Andrew raises in the research paper — are languages like C# 4 and Java that have nominal subtyping and contravariant conversions decidable? — has been answered; they are not. I am very pleased to have played an extremely minor role in that. It went down like this.

A “decision problem” is a problem with a “yes or no” answer. “Is this string a substring of that string?” or “Is there a cycle on this graph that visits every node exactly once?”  A problem is said to be “decidable” if it is possible to write a computer program that always answers the question in finite time, in theory. The problems I just gave are decidable. “Does this program eventually halt when given this input or does it infinite loop?” is famously undecidable. There are many problems whose decidability we do not yet know, but the standard technique is to show a way to transform one problem into another whose decidability is known.

This 2013 post asks for a collection of decision problems not known to be decidable or undecidable, and I answered that the question “is this type convertible to that type?” in a language with nominal subtyping and contravariant conversions was not known to be decidable or undecidable. A reader noticed that post, solved the problem, and wrote a short paper describing the proof. It turns out you can make the compiler simulate an arbitrary Turing machine encoded it in the type system, and therefore, convertibility is the same class of problems as the halting problem mentioned above: it is undecidable.

This post generated a number of reader responses back in 2008; “expression too complex is an understatement” summed up a lot of them.

Some readers did experiments and discovered that small variations on my examples could cause Visual Studio to crash; at the time, the IDE used a different type analyzer than the compiler, and there were a lot of weird bugs like that. Eliminating the duplicated analysis was a major focus for our work in C# 3. We had to add so many new analysis algorithms to make LINQ work, and we did not want to replicate them in the IDE. So we not only had a lot of feature work to do, but also a lot of architecture work to de-silo the compiler/IDE divide. Fun times.

Many people also took this opportunity to request return type covariance, which after only twenty years of it being requested, may actually make it into C# 9.

 

How to not get a question answered

Posted on by
1

Raymond has had lots of great posts over the years on how to not get a question answered. Some of the ways he points out that help ensure that a question goes unanswered are:

I would add that phrasing the question in terms of the attempted solution rather than the underlying problem is a great way to have a question remain unanswered. But I already did that. Today’s rant, rather, is:

A great way to not get your question answered is to treat the person that you’re asking to share their time and knowledge like they’re not a human being.

I understand that this is easy to do.

I understand that when someone is asking a technical question, it is almost always because they’re deeply frustrated by their inability to make some important specific complicated thing work the way they’d expect.

I understand that software makers are a big source of frustration in a more general sense. We all use imperfect software every day; little frustrations mentally accumulate in a way that little victories do not.

I understand that programming languages and software in general is often counterintuitive, and that understanding the complex reasoning behind a counterintuitive result is often seemingly of little practical import compared to just getting the problem at hand solved.

I understand that it is very easy to use an email client as an impersonal and automatic information resource something like a search engine, where you just type in your query, you get your result back, and you move on with your day.

I understand that there is an inherent and pervasive bias in pure-text communication which makes statements intended to be good-humoured sound sophomoric, makes statements which were intended to be friendly sound smarmy, makes statements which were intended to be enthusiastic sound brash, makes statements intended to be helpful sound condescending, makes statements which were intended to be precise and accurate sound brusque and pedantic, makes statements which were intended to be positive sound neutral, and makes statements which were intended to be neutral seem downright hostile.

Boy, do I ever understand that. For over four years I have deliberately tried to pitch the tone of this blog as good-humoured, friendly, enthusiastic, helpful, precise, accurate and positive; I suspect that most of the time I fail badly at most or all of those. Writing is hard.

I understand all that, from both sides; I’ve certainly been on both the sending and receiving ends of all of the above, many times.

Which is why I try very hard to be helpful to the complete strangers who send me overtly rude and frequently profane emails capped off with a request for me to to take time out of my day to explain something to them.

I try to see an upset or confused user as an opportunity to make a good impression; sometimes people do a complete 180 when you help them out and are appreciative and grateful. Sometimes they even send an unsolicited note to your manager, which is always pleasant.

But more often, I never hear from them again after their question is answered.

None of this is in their own best interests. It makes the human information sources they rely upon less likely to help them now and in the future.

So, just a friendly reminder. (A friendly, good-humoured, helpful, enthusiastic, positive reminder!) The people you are sending email to about your technical problems are people. It would be smart to briefly introduce yourself, describe your problem without insulting the people who created it, indicate what you’ve done to attempt to solve it yourself, actually ask for help, and later say thank-you for their time whether they manage to actually help you or not.

It’s smart because doing so is in your own best interests.

It would also be nice, but I actually am not particularly concerned about “nice” today. Nice is nice to have I suppose. But not being smart is just dumb.

Commentary from 2021:

Obviously this rant was born of frustration; one of my predecessors on the C# team, Anson Horton, recently told a story of the early days of answering C# questions internally at Microsoft that reminded me of this article. His story ended well but most of them did not. And of course it was similar answering JavaScript questions internally; it had nothing to do with C# as C#. The underlying problem was how Microsofties (and their customers!) were socialized to treat their colleagues with contempt, and not the software systems which facilitated it.

There were a lot of good comments posted on the original article; to summarize a few of them here:

  • Commiseration from people whose time had also been imposed upon by people asking rude and often unanswerable questions.
  • Some people said that this was post-it-note wisdom, and that was a compliment: they put a post-it note on their monitor that said “little frustrations accumulate, little victories do not”.
  • More ranting from me; I pointed out in a follow-up comment that there were entire multi-week stretches when I worked on subtle problems in method type inference for eight hours a day, and that maybe if I don’t answer your question on that subject within five minutes, there’s a reason for that. It takes research to answer a question correctly, and research takes time, and must be prioritized against many other important concerns.
  • A number of readers felt called out, and said yeah, sorry I never thanked you for solving my problem. One of which was a question I answered ten years previously! You’re welcome!

I was expressing some frustration here; I was not fishing for compliments or for thanks, but hey, I’ll take both if they’re offered.

 

Why Do Initializers Run In The Opposite Order As Constructors? Part Two

Posted on by
Reply

As you might have figured out, the answer to last week’s puzzle is “if the constructors and initializers run in their actual order then an initialized readonly field of reference type is guaranteed to be non null in any possible call. That guarantee cannot be met if the initializers run in the expected order.”

Suppose counterfactually that initializers ran in the expected order, that is, derived class initializers run after the base class constructor body. Consider the following pathological cases:

class Base
{
  public static ThreadsafeCollection t = new ThreadsafeCollection();
  public Base()
  {
    Console.WriteLine("Base constructor");
    if (this is Derived) 
      (this as Derived).DoIt();
    // would deref null if we are constructing an instance of Derived
    Blah();
    // would deref null if we are constructing an instance of MoreDerived
    t.Add(this);
    // would deref null if another thread calls Base.t.GetLatest().Blah();
    // before derived constructor runs
  }
  public virtual void Blah() { }
}
class Derived : Base
{
  readonly Foo derivedFoo = new Foo("Derived initializer");
  public DoIt()
  {
    derivedFoo.Bar();
  }
}
class MoreDerived : Derived
{
  public override void Blah() 
  { 
    DoIt(); 
  }
}

Calling methods on derived types from constructors is a bad idea but it is not illegal. And stuffing not-quite-constructed objects into global state is risky, but not illegal. I’m not recommending that you do any of these things — please, do not, for the good of us all. I’m saying that it would be really nice if we could give you an ironclad guarantee that an initialized readonly field is always observed in its initialized state, and we cannot make that guarantee unless we run all the initializers first, and then all of the constructor bodies.

Note that of course if you initialize your readonly fields in the constructor then all bets are off. We make no guarantees as to the fields not being accessed before the constructor bodies run.

Why Do Initializers Run In The Opposite Order As Constructors? Part One

Posted on by
Reply

Pop quiz!

What do you expect the output of this program to be?

using System;
class Foo
{
  public Foo(string s)
  {
    Console.WriteLine("Foo constructor: {0}", s);
  }
  public void Bar() { }
}
class Base
{
  readonly Foo baseFoo = new Foo("Base initializer");
  public Base()
  {
    Console.WriteLine("Base constructor");
  }
}
class Derived : Base
{
  readonly Foo derivedFoo = new Foo("Derived initializer");
  public Derived()
  {
    Console.WriteLine("Derived constructor");
  }
}
static class Program
{
  static void Main()
  {
    new Derived();
  }
}

I got a question from a user recently noting that the order was not as they expected. One naively expects that the order will go “base initializers, base constructor body, derived initializers, derived constructor body”. In fact the order actually is that first all the initializers run in order from derived to base, and then all the constructor bodies run in order from base to derived.

The latter bit makes perfect sense; the more derived constructors may rely upon state initialized by the less derived constructors, so the constructors should run in order from base to derived. But most people assume that the call sequence of the code above is equivalent to this pseudocode:

// Expected
BaseConstructor()
{
  ObjectConstructor();
  baseFoo = new Foo("Base initializer");
  Console.WriteLine("Base constructor");
}
DerivedConstructor()
{
  BaseConstructor();
  derivedFoo = new Foo("Derived initializer");
  Console.WriteLine("Derived constructor");
}

When in point of fact it is equivalent to this:

// Actual
BaseConstructor()
{
  baseFoo = new Foo("Base initializer");
  ObjectConstructor();
  Console.WriteLine("Base constructor");
}
DerivedConstructor()
{
  derivedFoo = new Foo("Derived initializer");
  BaseConstructor();
  Console.WriteLine("Derived constructor");
}

That explains the mechanism whereby the initializers run in order from derived to base and the constructor bodies run in the opposite order, but why did we choose to implement that mechanism instead of the more intuitively obvious former way?

Puzzle that one over for a bit, and then read on for a hint.

…

…

…

…

…

…

…

…

The “readonly” modifiers in there were no accident. The code gives the appearance that any call to derivedFoo.Bar() and baseFoo.Bar() should never fail with a null dereference exception because both are readonly fields initialized to non-null values.

  1. Is that appearance accurate, or misleading?
  2. Now suppose initializers ran in the “expected” order and answer question (1) again.

I’ll post the answers and analysis next week. Have a fabulous weekend!