This is part one of a two-part series. Part two is here.
Can you find a lambda expression that can be implicitly converted to Func<T> for any possible T?
Continue reading
This is part one of a two-part series. Part two is here.
Can you find a lambda expression that can be implicitly converted to Func<T> for any possible T?
Continue reading
Occasionally when I’m debugging the compiler or responding to a user question I’ll need to quickly take apart the bits of a double-precision floating point number. Doing so is a bit of a pain, so I’ve whipped up some quick code that takes a double and tells you all the salient facts about it. I present it here, should you have any use for it yourself.[1. Note that this code was built for comfort, not speed; it is more than fast enough for my purposes so I’ve spent zero time optimizing it.]
No one I know at Microsoft asks those godawful “lateral-thinking puzzle” interview questions anymore. Maybe someone still does, I don’t know. But rumour has it that a lot of companies are still following the Microsoft lead from the 1990s in their interviews. In that tradition, I present a sequel to Keith Michaels’ 2003 exercise in counterfactual reasoning. Once more, we dare to ask the question “how well would the late Nobel-Prize-winning physicist Dr. Richard P. Feynman do in a technical interview at a software company?”
“Can a property or method really be marked as both abstract and override?” one of my coworkers just asked me. My initial gut response was “of course not!” but as it turns out, the Roslyn codebase itself has a property getter marked as both abstract and override. (Which is why they were asking in the first place.)
I thought about it a bit more and reconsidered. This pattern is quite rare, but it is perfectly legal and even sensible. The way it came about in our codebase is that we have a large, very complex type hierarchy used to represent many different concepts in the compiler. Let’s call it “Thingy”:
abstract class Thingy
{
public virtual string Name { get { return ""; } }
...
There are going to be a lot of subtypes of Thingy, and almost all of them will have an empty string for their name. Or null, or whatever; the point is not what exactly the value is, but rather that there is a sensible default name for almost everything in this enormous type hierarchy.
However, there is another abstract kind of Thingy, a FrobThingy, which always has a non-empty name. In order to prevent derived classes of FrobThingy from accidentally using the default implementation from the base class, we said:
abstract class FrobThingy : Thingy
{
public abstract override string Name { get; } }
...
Now if you make a derived class BigFrobThingy, you know that you have to provide an implementation of Name for it because it will not compile if you don’t.
Here’s a pattern you see all the time in C#:
class Frob : IComparable<Frob>
At first glance you might ask yourself why this is not a “circular” definition; after all, you’re not allowed to say class Frob : Frob(*). However, upon deeper reflection that makes perfect sense; a Frob is something that can be compared to another Frob. There’s not actually a real circularity there.
This pattern can be genericized further:
class SortedList<T> where T : IComparable<T>
Again, it might seem a bit circular to say that T is constrained to something that is in terms of T, but actually this is just the same as before. T is constrained to be something that can be compared to T. Frob is a legal type argument for a SortedList because one Frob can be compared to another Frob.
But this really hurts my brain:
class Blah<T> where T : Blah<T>
That appears to be circular in (at least) two ways. Is this really legal?
Yes it is legal, and it does have some legitimate uses. I see this pattern rather a lot(**). However, I personally don’t like it and I discourage its use.
This is a C# variation on what’s called the Curiously Recurring Template Pattern in C++, and I will leave it to my betters to explain its uses in that language. Essentially the pattern in C# is an attempt to enforce the usage of the CRTP.
So, why would you want to do that, and why do I object to it?
One reason why people want to do this is to enforce a particular constraint in a type hierarchy. Suppose we have
abstract class Animal
{
public virtual void MakeFriends(Animal animal);
}
But that means that a Cat can make friends with a Dog, and that would be a crisis of Biblical proportions! (***) What we want to say is
abstract class Animal
{
public virtual void MakeFriends(THISTYPE animal);
}
so that when Cat overrides MakeFriends, it can only override it with Cat.
Now, that immediately presents a problem in that we’ve just violated the Liskov Substitution Principle. We can no longer call a method on a variable of the abstract base type and have any confidence that type safety is maintained. Variance on formal parameter types has to be contravariance, not covariance, for it to be typesafe. And moreover, we simply don’t have that feature in the CLR type system.
But you can get close with the curious pattern:
abstract class Animal<T> where T : Animal<T>
{
public virtual void MakeFriends(T animal);
}
class Cat : Animal<Cat>
{
public override void MakeFriends(Cat cat) {}
}
and hey, we haven’t violated the LSP and we have guaranteed that a Cat can only make friends with a Cat. Beautiful.
Wait a minute… did we really guarantee that?
class EvilDog : Animal<Cat>
{
public override void MakeFriends(Cat cat) { }
}
We have not guaranteed that a Cat can only make friends with a Cat; an EvilDog can make friends with a Cat too. The constraint only enforces that the type argument to Animal be good; how you use the resulting valid type is entirely up to you. You can use it for a base type of something else if you wish.
So that’s one good reason to avoid this pattern: because it doesn’t actually enforce the constraint you think it does. Everyone has to play along and agree that they’ll use the curiously recurring pattern the way it was intended to be used, rather than the evil dog way that it can be used.
The second reason to avoid this is simply because it bakes the noodle of anyone who reads the code. When I see List<Giraffe> I have a very clear idea of what the relationship is between the List<> part — it means that there are going to be operations that add and remove things — and the Giraffe part — those operations are going to be on giraffes. When I see FuturesContract<T> where T : LegalPolicy I understand that this type is intended to model a legal contract about a transaction in the future which has some particular controlling legal policy. But when I read Blah<T> where T : Blah I have no intuitive idea of what the intended relationship is between Blah<T> and any particular T. It seems like an abuse of a mechanism rather than the modeling of a concept from the program’s “business domain”.
All that said, in practice there are times when using this pattern really does pragmatically solve problems in ways that are hard to model otherwise in C#; it allows you to do a bit of an end-run around the fact that we don’t have covariant return types on virtual methods, and other shortcomings of the type system. That it does so in a manner that does not, strictly speaking, enforce every constraint you might like is unfortunate, but in realistic code, usually not a problem that prevents shipping the product.
My advice is to think very hard before you implement this sort of curious pattern in C#; do the benefits to the customer really outweigh the costs associated with the mental burden you’re placing on the code maintainers?
(*) Due to an unintentional omission, some past editions of the C# specification actually did not say that this was illegal! However, the compiler has always enforced it. In fact, the compiler has over-enforced it, sometimes accidentally catching non-cycles and marking them as cycles.
(**) Most frequently in emails asking “is this really legal?”
(***) Mass hysteria!
One more easy one. I want to “sort” a list into a random, shuffled order. I can do that by simply randomizing whether any two elements are greater than, less than, or equal to each other:
myList.Sort((x, y) => (new Random()).Next(-1, 2));
That generates a random -1, 0 or 1 for every comparison, right? So it will sort the list into random order, right?
.
.
.
.
.
.
.
There are multiple defects here. First off, clearly this violates all our rules for comparison functions. It does not produce a total ordering, and in fact it can tell you that two particular elements are equal and then later tell you that they have become unequal. The sort algorithm is allowed to go into infinite loops or crash horribly when given such an ill-behaved comparison function. And in fact, some implementations of sorting algorithms attempt to detect this error and will throw an exception if they determine that the comparison is inconsistent over time.
Second, every time this thing is called it creates a new Random instance seeded to the current time, and therefore it produces the same result over and over again if called multiple times in the same timeslice; hardly random.
Shuffling is not sorting; it is the opposite of sorting, so don’t use a sort algorithm to shuffle. There are lots of efficient shuffle algorithms that are easy to implement. (That said, it is legal to shuffle by sorting a list ordered by a randomly chosen key. But the key must be chosen exactly once for each item in the list and there must be a correct comparison function on that key.)
Did you notice how last time my length comparison on strings was unnecessarily verbose? I could have written it like this:
static int ByLength(string x, string y)
{
if (x == null && y == null) return 0;
if (x == null) return -1;
if (y == null) return 1;
return CompareInts(x.Length, y.Length);
}
static int CompareInts(int x, int y) {
// positive if x is larger
// negative if y is larger
// zero if equal
return x - y;
}
static Comparison<T> ThenBy<T>(this Comparison<T> firstBy, Comparison<T> thenBy)
{
return (x,y)=>
{
int result = firstBy(x, y);
return result != 0 ? result : thenBy(x, y);
}
}
Much nicer! My string length comparison method is greatly simplified, I can compose comparisons easily with the ThenBy extension method, and I can reuse CompareInts as a helper function in other comparisons I might want to write.
What’s the defect now?
.
.
.
.
.
.
This one should be easy after last time. The lengths of strings are always positive integers and in practice they never go above a few hundred million; the CLR does not allow you to allocate truly enormous multi-gigabyte strings. But though CompareInts is safe for inputs which are string lengths, it is not safe in general. In particular, for the inputs Int32.MinValue and Int32.MaxValue, the difference is 1. Clearly the smallest possible integer is smaller than the largest possible integer, but this method gives the opposite result! CompareInts should read:
static int CompareInts(int x, int y)
{
if (x > y) return 1;
if (x < y) return -1;
return 0;
}
The moral of the story here is that a comparison function that doesn’t compare something is probably wrong. Subtraction is not comparison.
Next time on FAIC: One more bad comparison.
Suppose I want to sort a bunch of strings into order first by length, and then, once they are sorted by length, sort each group that is the same length by some other comparison. We can easily build such a device with higher-order programming:
static Comparison<string> FirstByLength(Comparison<string> thenBy)
{
return (string x, string y) =>
{
// Null strings are sorted before zero-length strings; remember, we need to provide a total ordering.
if (x == null && y == null)
return 0;
if (x == null)
return -1;
if (y == null)
return 1;
if (x.Length > y.Length)
return 1;
if (x.Length < y.Length)
return -1;
// They are the same length; sort on some other criterion.
return thenBy(x, y);
};
}
Super. This idea of composing new comparison functions out of old ones is pretty neat. We can even built a reversal device:
static Comparison<string> Reverse(Comparison<string> comparison)
{
return (string x, string y) => -comparison(x, y);
}
Something is subtly wrong in at least one of these comparison functions. Where’s the defect?
.
.
.
.
.
.
.
.
Let’s restate that contract again. The comparison function returns a negative integer if the first argument is smaller than the second, a positive integer if the first is greater than the second, and zero if they are equal. Any negative integer will do, and in particular, Int32.MinValue is a negative integer. Suppose we have a bizarre comparison function that returns Int32.MinValue instead of -1:
Comparison<string> bizarre = whatever;
and we compose it:
Comparison<string> reverseFirstByLength = Reverse(FirstByLength(bizarre));
Suppose two strings are equal in length and bizarre returns Int32.MinValue for those strings. Reverse should return a positive number, but -Int32.MinValue either throws an exception (in a checked context) or returns Int32.MinValue right back (in an unchecked context). Remember, there are more negative numbers that fit into an integer than positive numbers, by one.
The right implementation of Reverse is either to spell it out:
static Comparison<string> Reverse(Comparison<string> comparison)
{
return (string x, string y) =>
{
int result = comparison(x, y);
if (result > 0) return -1;
if (result < 0) return 1;
return 0;
};
}
Or to simply swap left and right:
static Comparison<string> Reverse(Comparison<string> comparison)
{
return (string x, string y) => comparison(y, x);
}
Next time on FAIC: Another way that comparisons go wrong.
The mutable List<T> class provides an in-place sort method which can take a comparison delegate. It’s quite handy to be able to sort a list into order by being able to compare any two elements, but you have to make sure you get it right.
First off, what are the requirements of the comparison delegate? They are clearly documented: the comparison takes two elements and returns a 32 bit signed integer. If the first element is greater than the second then the integer is greater than zero. If the first element is less than the second then the integer is less than zero. If the first element is equal to the second then the integer is zero.
See if you can figure out why each of these comparisons can give bad results.
Comparison #1: Putting on my top hat:
enum Clothes
{
Hat,
Tie,
Socks,
Pocketwatch,
Vest,
Shirt,
Shoes,
Cufflinks,
Gloves,
Tailcoat,
Underpants,
Trousers
}
static int Compare(Clothes x, Clothes y)
{
const int xGreater = 1;
const int yGreater = -1;
// If x has to go on after y then x is the greater
// If y has to go on after x then y is the greater
// Otherwise, they are equal
switch (x)
{
case Clothes.Tie:
if (y == Clothes.Shirt) return xGreater;
break;
case Clothes.Socks:
if (y == Clothes.Shoes) return yGreater;
break;
case Clothes.Pocketwatch:
if (y == Clothes.Shirt || y == Clothes.Vest) return xGreater;
break;
case Clothes.Vest:
if (y == Clothes.Shirt) return xGreater;
if (y == Clothes.Tailcoat || y == Clothes.Pocketwatch) return yGreater;
break;
case Clothes.Shirt:
if (y == Clothes.Tie || y == Clothes.Pocketwatch ||
y == Clothes.Vest || y == Clothes.Cufflinks || y == Clothes.Tailcoat)
return yGreater;
break;
case Clothes.Shoes:
if (y == Clothes.Trousers || y == Clothes.Socks || y == Clothes.Underpants)
return xGreater;
break;
case Clothes.Cufflinks:
if (y == Clothes.Shirt) return xGreater;
break;
case Clothes.Tailcoat:
if (y == Clothes.Vest || y == Clothes.Shirt) return xGreater;
break;
case Clothes.Underpants:
if (y == Clothes.Trousers || y == Clothes.Shoes) return yGreater;
break;
case Clothes.Trousers:
if (y == Clothes.Underpants) return xGreater;
if (y == Clothes.Shoes) return yGreater;
break;
}
return 0;
}
OK, before you read on, can you figure out what the defect here is? It seems perfectly straightforward: if two things have to be ordered then they are ordered, and if not, then we say we don’t care by setting them equal.
Does that work?
.
.
.
.
.
.
.
If you actually try it out on a real list of randomly ordered clothes, you’ll find that much of the time this sorts the list into a senseless order, not into an order that preserves nice properties like shoes going on after trousers. Why?
An undocumented but extremely important assumption of the sorting algorithm is that the comparison function provides a consistent, total order. Part of providing a total order means that the comparison must preserve the invariant that things equal to the same are equal to each other. This idea that if you don’t care about the order of two things then you can call them equal is simply false. In our example Tie is equal to Hat and Hat is equal to Shirt, and therefore the sort algorithm is justified in believing that Tie should be equal to Shirt, but it isn’t.
Suppose, for example, the first element is Hat. A sort algorithm is perfectly justified in scanning the entire list, determining that everything is equal to the first element, and concluding that therefore it must already be sorted. Clearly a list where every element is equal to the first element is sorted! The actual implementation of QuickSort in the BCL is quite complex and has several clever tricks in it that help optimize the algorithm for common cases, such as subsets of the list already being in sorted order. If the comparison is not consistent then it is easy to fool those heuristics into doing the wrong thing. And in fact, some sort algorithms will go into infinite loops or crash horribly if you give them an incomplete comparison function.
The right algorithm for sorting a set with a partial order is topological sort; use the right tool for the job.
Next time on FAIC: we do the same thing, backwards
Suppose you use an anonymous type in C#:
var x = new { A = "hello", B = 123.456 };
Ever taken a look at what code is generated for that thing? If you crack open the assembly with ILDASM or some other tool, you’ll see this mess in the top-level type definitions
.class '<>f__AnonymousType0`2'<'<A>j__TPar','<B>j__TPar'>
What the heck? Let’s clean that up a bit. We’ve mangled the names so that you are guaranteed that you cannot possibly accidentally use this thing “as is” from C#. Turning the mangled names back into regular names, and giving you the declaration and some of the body of the class in C#, that would look like:
[CompilerGenerated]
internal sealed class Anon0<TA, TB>
{
private readonly TA a;
private readonly TB b;
public TA A { get { return this.a; } }
public TB B { get { return this.b; } }
public Anon0(TA a, TB b)
{ this.a = a; this.b = b; }
// plus implementations of Equals, GetHashCode and ToString
}
And then at the usage site, that is compiled as:
var x = new Anon0<string, double>("hello", 123.456);
Again, what the heck? Why isn’t this generated as something perfectly straightforward, like:
[CompilerGenerated]
internal sealed class Anon0
{
private readonly string a;
private readonly double b;
public string A { get { return this.a; } }
public double B { get { return this.b; } }
public Anon0(string a, double b)
{ this.a = a; this.b = b; }
// plus implementations of Equals, GetHashCode and ToString
}
Good question. Consider the following. Suppose you have a library assembly, not written by you, that contains the following types:
public class B
{
protected class P {}
}
Now, in your source code you have:
class D1 : B
{
void M()
{
var x = new { P = new B.P() };
}
}
class D2 : B
{
void M()
{
var x = new { P = new B.P() };
}
}
We need to generate an anonymous type, or types, somewhere. Suppose we decide that we want the two anonymous types – which have the same types and the same property names – to unify into one type. (We desire anonymous types that are structurally identical to unify within an assembly because that enables scenarios where multiple methods use generic type inference to infer the same anonymous type; you want to be able to pass instances of that anonymous type around between such methods. Perhaps I’ll do an example of that in the new year.)
Where do we generate that type? How about inside D1:
class D1 : B
{
[CompilerGenerated]
??? sealed class Anon0 { public P P { get { ... } } ... }
void M()
{
var x = new { P = new B.P() };
}
}
What is the desired accessibilty of Anon0 in the location marked with question marks? It cannot be private or protected, because then D2 cannot see it. It cannot be either public or internal, because then you’d have a public/internal type with a public property that exposes a protected type, which is illegal. Nor can it be either “protected and internal” or “protected or internal” by similar logic. It cannot have any accessibility! Therefore the anonymous type cannot go in D1. Obviously by identical logic it cannot go in D2. It cannot go in B because you don’t have the ability to modify the assembly containing class B. The only remaining place it can go is in the global namespace. But at the top level an internal type cannot refer to P, a protected nested type. P is only accessible inside a derived class of B but we’ve already ruled all those out.
But we can put the anonymous type at the top level if it never actually refers to P. If we make generic class Anon0<TP> and construct it with P for TP, then P only ever appears inside D1 and D2, and yet the types unify as desired.
Rather than coming up with some weird heuristic that determined when anonymous types needed to be generic, and making them normally typed otherwise, we simply decided to embrace the general solution. Anonymous types are always generated as generic types even when doing so is not strictly necessary. We did extensive performance testing to ensure that this choice did not adversely impact realistic scenarios, and as it turned out, the CLR is really quite buff when it comes to construction of generic types with lots of type parameters.
And with that, I’m off for the rest of the year. Air travel is too expensive this year, so I’m going to miss my traditional family Boxing Day celebration, but I’m sure it’ll be delightful to spend some time in Seattle for the holidays. I hope you all have a safe and festive holiday season, and we’ll see you for more fabulous adventures in 2011.