In Code review as human alignment, in the era of LLMs, I talked about how we should approach code review with other team members, where you have a pre-existing relationship and care about building human-to-human alignment over time. However…
How does PyTorch autograd deal with mutation? In particular, what happens when a mutation occurs on a view, which aliases with some other tensor? In 2017, Sam Gross implemented support for in-place operations on views, but the details of wh…
I was coaching some juniors on effective use of AI coding agents (like, looking over their shoulder as they were prompting the LLMs) and one reoccurring theme was that the AI agents were demanding a lot more of their reading code skills, co…
Previously in AI-assisted programming for spmd_types, I mentioned that I have been enjoying using Sapling (the version control system Meta uses internally) to manage parallel agents on worktrees. In this post, I want to describe this workfl…
spmd_types is a type checker for distributed PyTorch programs. I’ll write a blog post about the system itself as it gets closer to completion, but in this post I want to talk about my workflow for AI-assisted programming in the project. In
A silent BC breaking change is a change of semantics in an API that is not immediately obvious; e.g., the old usages of the broken behavior won’t raise a compile or runtime error on upgrade. If you are a project that cares about backwards c…
A silent BC breaking change is a change of semantics in an API that is not immediately obvious; e.g., the old usages of the broken behavior won’t raise a compile or runtime error on upgrade. If you are a project that cares about backwards c…
I recently received the following question about vibe-coding for tlparse, a structured log parser for torch.compile (slightly edited for ease of reading): Hi Ed, I have some thoughts on vibe-coding tlparse and I’m curious what your opinions…
I recently received the following question about vibe-coding for tlparse, a structured log parser for torch.compile (slightly edited for ease of reading): Hi Ed, I have some thoughts on vibe-coding tlparse and I’m curious what your opinions…
A central thesis of sharding in types is that the backward sharding can be directly computed from the forward sharding. This is not true for DTensor today, e.g., as seen in sum to Partial, and it leads to confusion where users cannot easily…
DTensor has famously terrible eager mode performance; for example, this paper measured a 35-60% slowdown in end-to-end training performance with and without DTensor (with DTensor operations taking at least 7x longer than actually running th…
Conventionally, a type system is something that classifies values into data types like float32 or int64. However, fancy type systems go beyond data types, allowing us to talk about potentially arbitrary invariants on data; for example, if w…