It’s been almost two years since my last update here. A lot has happened. I hope you all are continuing to weather the ongoing multiple global pandemics and other anthropogenic crises.
Apologies that this is so long; I didn’t have time to make it shorter.
Obviously blogs do not come with a service level agreement, but some explanation is in order for the long gap. It’s pretty simple.
- Facebook actively discourages people to blog about their work.
- Working from home during the covid pandemic was enervating.
- After thinking about programming languages for many hours a week for over 25 years, I simply didn’t have the energy and enthusiasm to write much on the subject.
Blogging was an easy thing to drop in favor of pursuits that got me away from sitting in front of a screen in my home office. I’ve been spending my leisure time in the last couple years working on improving my nature photography skills and learning to scuba dive. Turns out you cannot catch covid when below 15 metres of seawater. And there are weird slugs in the Puget Sound!
Photos of the author and a golden dirona nudibranch by Amber, who convinced me to take up diving.
Today is the tenth anniversary of moving my blog to ericlippert.com on my last day at Microsoft, the fiftieth anniversary of my birth, and my last day at Facebook-now-Meta.
My team — Probabilistic Programming Languages — and indeed entire “Probability” division were laid off a couple weeks ago; the last three years of my work will be permanently abandoned.
The mission of the Probability division was to create small teams that applied the latest academic research to real-world at-scale problems, in order to improve other groups’ decision-making and lower their costs. New sub-teams were constantly formed; if they didn’t show results quickly then they were failed-fast; if they did show results then they were reorganized into whatever division they could most effectively lower costs.
We were very successful at this. The PPL team in particular was at the point where we were regularly putting models into production that on net reduced costs by millions of dollars a year over the cost of the work. We were almost ready to be spun off.
We foolishly thought that we would naturally be protected from any layoffs, being a team that reduced costs of any team we partnered with. In retrospect, that was a little naive. A team that reduces costs of other teams is not on anyone’s critical path.
The whole Probability division was laid off as a cost-cutting measure. I have no explanation for how this was justified and I note that if the company were actually serious about cost-cutting, they would have grown our team, not destroyed it.
Speaking of cutting costs, the company is still pouring multiple billions of dollars into vaporware called “the metaverse”. News flash: no one wants to wear VR goggles to spend any time in a digital heaven where the role of God is played by Mark Zuckerberg and you can do anything you can imagine, including “work” and “shop”.
Ugh.
I would be happy to be shown to be wrong, wrong, wrong. Maybe there is a useful, engaging, fun, just, equitable, democratic, sustainable, novel VR experience where the avatars have legs, but Meta is $20 billion in and aside from the legs I don’t see any evidence that any of the above is forthcoming.
Yes, I am a little vexed.
I have a great many people to thank for my time at Facebook: Erik Meijer for recruiting me and finding seven years worth of interesting problems for me to dig into. Peter Hallam, with whom I have now worked with on three compiler teams at three companies, for encouraging me to take that offer. Walid Taha, Michael Tingley, John Myles White, Joe Pamer and Satish Chandra for their leadership and mentorship. And to many, many coworkers too numerous to mention here. The quality of the people I worked with at Facebook was amazing. Everyone was kind, smart, dedicated, thoughtful, generous with their time and knowledge, and a genuine pleasure to work with. I learned so much from all of them. Leaving those teammates is the hardest part.
Lots of people have asked how they can help me and my team. I am so grateful and appreciative. Friends, colleagues, strangers on Twitter, just about everyone has been sympathetic and helpful. Most of my team has found other positions and I am hopeful that the rest will soon.
I am not looking for another position at this time.
Why’s that?
I know I don’t look it, but I’m beginning to feel it in my heart. I feel thin, sort of stretched, like butter scraped over too much bread. I need a holiday. A very long holiday. And I don’t expect I shall return. In fact I mean not to.
I am very fortunate to have spent the pandemic thus far working safely from home, for a supportive team and for excellent pay. But after >26 years of thinking about programming languages for corporations, and the last three years of my work being thrown away, I need a good long corporate detox before I go looking again.
Coming up next on FAIC:
The work we did on Bean Machine, our embedded Python DSL for Bayesian inference, is quite interesting. In coming episodes I’ll explain what it is, how it works, and what we learned. No one else is ever going to do this post-mortem analysis, so I might as well!
Fabulous adventures in coding 
Welcome back! I missed reading your blog. I hope you’re doing okay in these crazy times Eric.
Thanks! I am housed and fed and healthy; I’m doing fine.
Thank you for the update, and the nifty photos! I’m not sure what specifically to say to be encouraging and sympathetic (just ask my wife about how bad I am at being encouraging), but I hope you continue to have (and share about) fabulous adventures, whether coding or otherwise.
Thanks; I plan to!
Happy to see you blogging again. I love to read your posts. I am sorry to hear about the lay-off and the reason for it, but I hope you get to take off as much time as you need to move on to something else.
Thanks; I’m glad to be back too. Looking forward to some time off.
It’s disappointing to hear of so much work thrown away, but I’m glad to see you return to writing publicly! I hope you’re doing well otherwise. I’ve always loved your writing around the net, and can’t wait to read more.
Thanks! I’m doing ok. Looking forward to having some time to write.
Welcome back! Even though I used the feedly aggregator, I’d often visit ericlippert.com to see if the internet was broken. Sorry to hear about vexing corporate decisions, but on the other hand, WTF (Welcome to Fifty)!
Thank you! Weird to be half a century old but I’ll take it.
It good to have you back Eric! Hopefully, you now have time to answer more questions on stackoverflow. Your help is badly needed since while questions are being answered (quick fix), most don’t have the very details that are required for a full picture. Painting those details is an art – and thus it requires a skillful artist.
Thanks! I still check in on SO occasionally but it is a different vibe than it was ten years ago. Much less fun, much less rewarding. The quality of the questions is pretty poor and there’s a lot of negativity from the people I try to help. So I’ve scaled that back pretty dramatically. If any of that changes I’d consider putting more time into it.
Same feeling on SO here…
Sorry to read programming languages were no longer your main focus and sad too 😦 so I watched https://www.youtube.com/watch?v=KzSY9WQXjzg again as a souvenir of the good old times 😉
Big fan of your’s writing.Welcome back.Love to read insightful writings
Thanks!
Thank you for giving news : I hoped you havent been ill with Covid. Sorry for your abandoned work, I hope that your researches could come to life in any kind !
(Sorry for my poor English, I’m French and really loved reading your blogs and stackov’ posts)
Your English is fine! Much better than my terrible high school Quebecois French. 🙂
Fortunately I have managed to avoid covid so far, by being up to date on vaccines, avoiding crowds and enclosed spaces with strangers, masking when I cannot avoid that, testing regularly before socializing, and massively reducing the amount of travel and dining out that I’d normally do in a year.
I’m not very happy about spending another few years doing that but it looks like I’m going to have to.
Thanks for your kind words!
Hey welcome back to the blog! Sorry to hear about the layoffs. For all the downsides and disappointments, it seems like it’s come at the right time – as you say, you can now take the time to rest and what a gift that is! All the best!
Thanks Rob! Good to hear from you. Would love to catch up with you and the family sometime.
I’m so, so sorry. I had two years of work killed by academic infighting and I wouldn’t wish that soul-crushing disappointment on anyone.
Thanks — it is indeed disappointing and is rather putting a damper on any celebrations today.
> I am not looking for another position at this time…. and the last three years of my work being thrown away, I need a good long corporate detox before I go looking again.
I really feel this. I’m about to finish up a contract working on a product that’s about to be killed before ever really seeing the light of day, and just when we were starting to get some good results. It’s kind of hard not to feel like Sisyphus at this point. I’m not really interested in looking for something else for a while, either.
Sorry to hear that! Lots of it going around apparently.
Sorry to hear you & your team have been through the wringer on this one, Eric. I’ve worked on projects that have been shelved and have also been laid off, and it can be deeply frustrating — especially when the work has tangible value.
Looking forward to seeing more blog entries, as your articles are always interesting. I was (gladly) corrected/educated by you on Stackoverflow a few times over the years, too! Always appreciated.
It is frustrating! I am looking forward to blogging more though. Thanks for the kind words.
I wasn’t able to join the probabilistic team after Bootcamp, and one of my biggest regrets was not trying to join it again before leaving Facebook/Meta.
Just curious: where would you start looking for interesting work? I’m finding it hard to find a job that genuinely interests me.
Well you’d be just looking for work now if you did — but it was fun while it lasted.
See the twitter thread I linked in the post; there are a lot of interesting opportunities there.
Enjoy your very well deserved corporate detox Eric – nice to see you blogging again, Meta’s loss will be all our gain! Best wishes from across the pond.
Thank you Chris, that’s kind of you to say. And I think I have never thanked you for your so-frequent support of my blog over the years. I really appreciate all the links. Thank you!
Sigh. Knowing that you were there working to make better languages for us all was the one redeeming thought that I had about FB. So much for that.
Pat, good to hear from you, it’s been too long.
There are still a lot of talented people working on Hack and various other projects, but yeah, the really innovative language work seems to have fallen by the wayside somewhat.
A most sincere welcome back Eric. Checking your blog every few days for the last couple of years had me between fearing for a worst case and hoping for the best case – that you were just having too much fun doing other things. Unfortunately it sounds like it was somewhere in-between.
I’m terribly sorry to hear that all your work the last few years may end up being for naught (I’ve been there and know it is crushing), and that your team was dismissed (the abject decisions a company makes to “cut costs” rarely make any sense). And it happening alongside the billions of dollars of metaverse nonsense undoubtedly make it even more frustrating.
However I will selfishly say that I’m delighted you’re back and looking forward to any future blogging you plan to do. And less-selfishly, very glad to hear that you’re doing well otherwise.
> Working from home during the covid pandemic was enervating.
I feel that in my bones.
Speaking of the “metaverse”, I’m surprised sometimes at the complete lack of credit given to Snow Crash (https://en.wikipedia.org/wiki/Snow_Crash), the novel that originated the term and idea. It’s somewhat interesting to me that one of the biggest companies can change their name and create a product based on a work of fiction, without the author or book getting any of the notice they deserve. I admit I’m curious if there’s anything about that you want to / can share personally or from a previous employee of Facebook.
Again, welcome back!
Thank you for those kind thoughts.
Regarding Snow Crash I can share a little anecdote. A year ago when the pivot to “meta” was announced, the first thing I said in my team group chat (context, I was then older than everyone else on the team by far) was “I’m old enough to have read Snow Crash the week it came out and IT WAS A DYSTOPIA, why are we building it?”
A number of my teammates had no idea that “metaverse” was coined in that novel.
Glad to see you posting again! Facebook sure didn’t understand the value of what they had with you and your team. And now I have one more thing I can hang on FB/Meta: depriving me of two years of interesting Eric Lippert posts due to their corporate policies. That’s two years I missed out on learning interesting stuff like more about probabilistic programming languages. Welcome back.
Thanks Eric, always a pleasure to hear from you, and we should get together and catch up sometime.
👍
I’m sorry Facebook-now-Meta punted your group like that. Layoffs are frustrating any time they happen, but pointless layoffs and throwing away so much work… that’s a really hard pill to swallow. I’m so sorry.
Selfishly, I’m glad to see you blogging again, and I hope you’ll post about photography and scuba too.
I saw your comment about Stack Overflow. It’s changed, yeah. And of course the company that owns it has changed a lot too. If you’re ever interested in Q&A on a smaller, friendlier scale, you might be interested in the software-development community on Codidact. Not trying to be spammy.
Thank you! I will check it out.
Eric wow wow wow!
Too much to respond to — so I’ll just say — best wishes.
I know you said “I am housed and fed and healthy; I’m doing fine.” … but if, on top of that, you also feel angry and empty and many other things, it’s certainly understandable.
It’s been a heck of a few weeks but I’m doing OK. Thanks!
Fascinating, thanks for the update.
Interesting to hear about “regularly putting models into production that on net reduced costs by millions of dollars a year over the cost of the work”. I wonder if the rest of the org was unable to grasp what these models were doing, considering recent reports:
«The documents describe data anarchy within the company, where people responsible for data systems are unaware of how other people in the company use their system. In some cases even the engineers using a system may not be able to understand what is happening because, according to a Meta engineer, ‘it is not possible for humans to understand’.»
https://www.iccl.ie/news/unsealed-court-documents-reveal-data-anarchy-at-meta/
Like most of these articles, there is a kernel of truth there but the full story is more complicated. (The Gell-Mann Amnesia principle strikes again.)
Facebook does an extremely good job of solving a particular privacy problem: automatic imposition of privacy-aware filters over queries against the social graph. That is, if you mark a post on FB as “friends only”, then any program that a FB employee writes to add a feature to the web site that manipulates that post will by default impose a filter that prevents that post from showing up in the results of a query executed on behalf of a non-friend.
FB does a relatively poor job of solving “purpose” privacy problems. If you give your phone number to FB for two-factor authentication, can FB use that phone number for any other purpose? In many countries, no, that’s a regulatory violation. But unlike social graph queries, raw SQL queries against the backing store historically did not automatically impose a purpose-based query filter; it was very easy to accidentally use data for a restricted purpose. Characterizing this technical problem as “data chaos” makes for snappy copy but it lacks nuance.
A few years ago I was briefly on loan to the team which is solving this problem; FB did an engineering post about it yesterday. https://engineering.fb.com/2022/11/30/data-infrastructure/static-analysis-sql-queries/
Welcome back! Like others that have commented it is sad that your work at Meta has been closed down, but I am also looking forward to more blogging from you.
Your series that ultimately revealed the async await functionality in C# was one of my favourites of all time.
Thank you! I’m looking forward to it as well.
Pingback: Dew Drop – December 1, 2022 (#3819) – Morning Dew by Alvin Ashcraft
> Facebook actively discourages people to blog about their work.
I’m wondering about this, can you elaborate? I know that there exists something akin to devblogs.microsoft.com: https://engineering.fb.com/
At Microsoft, developers were for a long time strongly encouraged to write in their own voice on their personal blogs about their work. FB employees are asked not to do that.
The engineering blog has a lot of interesting content but it is essentially a PR channel for the company. There’s nothing wrong with that; it’s a good channel. But it’s also not the same.
Discovered your blog and have enjoyed it. I’m sorry to hear about the fate of the Probability division, both for yourself and team as well as the lost opportunity for PPL success stories in industry. I’ve been working with or toying around with PPLs for a while now from the statistics side, so count me in as very interested in a deep dive on your Bean Machine choices.
Thanks! I’ll be looking at it from the compiler implementation side, not the statistics side, but I hope you enjoy it all the same. 🙂
That’s actually perfect, since that’s the side I’m both fascinated by and have no formal training 🙂 We played around with transformational compilation of probabilistic programs for things like conditioning, marginalization, other statistical operations, but working in Python made that difficult.
Of course you’d end with a Tolkien quote; I hope your holiday is as restful as Bilbo’s. Time for another read-through! I recently dug through your archives pulling out coding tips for my team; thanks for your many helpful posts over the years. Take care!
Thanks! I am looking forward to having time to get through the rather large stack of unread Tolkiana that has accumulated over the last few years.
Did you ever manage to apply ML to network congestion behaviors (bufferbloat?)
My team didn’t work on that problem but it is certainly amenable to this sort of analysis. We did do some work on a related problem of “given observations of dropped packets in a data center and a prior on the reliability of each router, which router in this data center is most likely to need fixing/replacing?”
I thank you for keep bending my mind all these years. Don’t kid yourself about staying away though :). I can hardly wait the next code adventures. But, till then, enjoy the sound of “silence”!!
Thank you!
Welcome back Erick, as you can see we’re still here, sorry to hear about your work being thrown away :(, thanks for sharing your story, enjoy the holidays
Having spent much of my career developing or helping develop brilliant technology that is figuratively rusting out in some back lot or moldering in some lawyer’s IP file drawer, I can feel your pain.
It is a deplorable part of the human condition that wealth aggregation is so decoupled from those of us who who actually do the creating, that those who control the wealth (hence hold the power) are OK with tossing our brilliant work into the dustbin. One’s craft, or art, or legacy, or whatever it is, deserves a disposition commensurate with its obvious value, even if the bosses can’t figure out how to monetize it.
Here and there, though, we might be fortunate enough to push something out the door that actually solves someone’s problem. Between that and the (occasionally) handsome pay, it’s marginally worthwhile, otherwise we would go off to to do something else. By that metric I think you have been successful. I hope you can keep up the good work!
Thanks for those kind words. It’s not even that the work itself is tossed; that’s happened to me plenty of times, it’s irritating, but I get how that happens. The painful thing is seeing the value of the whole team discarded, without any real reckoning of the human cost.
Glad to see you’re back blogging! I’ve always enjoyed reading what you have to say. Take care.
Thank you! I see you’re no longer at Microsoft; I’ve been meaning to check out what mobilize is up to; looks like interesting stuff!
My perhaps more-than-just-slightly jaundiced view is that the vast majority of talk about developer efficiency is lip service. At the end of the day management can’t measure productivity, so they can’t measure an improvement in it, and if they can’t explain it to _their_ boss, or their boss’ boss, with numbers then they will not get credit for it. Another factor is that many organizations are so obsessed with the latest shiny new thing that they really don’t care how well, or efficiently, they are producing the bread-and-butter.
Always good to hear from you Roger!
Though I take your point, the work of my team, probabilistic programming, was not in developer efficiency or productivity. Rather we were looking at business processes with clear metrics.
For example, we developed a model to determine when two human content reviewers disagree as to whether a post belongs on FB, which one is more likely correct. That not only saved money — a LOT more money than it cost, by not getting a third person break the tie — but it also improved content reviewing and exposed fewer human reviewers to toxic content.
Or for example, we developed a model to predict which of a set of experiments were most and least likely to produce a statistically significant result given the size of the control and treatment groups (a factor called “experimental power” by statisticians). Underpowered experiments are unreliable, overpowered experiments are expensive; the company both saves money and makes better decisions when power is allocated correctly, and we could put a bound on the likelihood that an experiment was powered correctly.
This is just a fraction of the sorts of tasks my team was undertaking when we were dissolved.
I’d like to take the occasion to thank you for all your writing on StackOverflow, your blog and in other places, freely available to the benefit of everyone. I have been following it for about twelve years now, and it has always been an invaluable source of insight and inspiration. Provided you’re still enjoying it, please continue.
Thanks for those kind words; I appreciate it! I’m looking forward to doing more writing in 2023.
I am sorry to hear that your team was treated so poorly. Now about my issues 🙂
While C# is mostly in a maintenance state, Microsoft could certainly use some help with fixing neglected corners of the language, my most painful one being the increasing lack of relevance of Expression trees to the rest of the language as it gets further behind. Perhaps returning to familiar ground would be an option?