The Amazing that is PD-1 Blockade

The office is clearly closing up shop for the holiday; this’ll be my last task before heading out myself for a few days. I actually really like these quiet days at work; I can often pick up a few of the long-running side projects that are so hard to squeeze in when everybody’s here. This week it’s been some integration work with Illumina’s BaseSpace service, which is honesty pretty sweet.

End of the year holiday time is also always good for a bit of reflection, and I’ve been appreciating the opportunity I’ve had to start working here with the folks at Adaptive. In particular, a paper was just published by a customer that just has me shaking my head at how incredible this space is.

PD-1 Blockade drugs like Nivolumab are the current face of immunology — these incredible therapies seem to be showing more success against cancers than anything we’ve tried in years. Rather than just trying to knock out tumor cells with radiation or toxic chemicals, PD-1 blockades unleash our own immune systems to do the job. Here’s the deal:

Our T- and B-Cells express a particular protein called “programmed cell death 1.” Like other receptors, PD-1 is anchored in the cell kind of like a blade of grass sits in the ground, with part of it inside the cell wall, and the rest dangling outside.

PD-1’s job is to slow down our immune response. It waits for special proteins (PD Ligands) to float by and bind to its receptor end, which can result in one of two behaviors. Normal immune cells just commit suicide; regulatory T-Cells do the opposite and get busy. The net effect is that when PD-1 is activated, your immune system starts to go quiet.

This is normally a good thing — for example, PD-1 keeps our immune system from attacking itself. But many types of cancer have “figured out” the game; they artificially accelerate the production of PD ligands themselves! This is amazing to think about — the cancers have actually evolved to suppress our own immune system so that we can’t fight back.

Once you see how this works, PD-1 blockade therapies seem pretty obvious — create a drug that stops PD-1 from binding with its ligands, and the immune system is freed up to go nuts. And, holy crap, this actually works!

But it’s also expensive, and it doesn’t always work. That’s where Adaptive comes in: we’ve now shown that the clonality* of a person’s immune repertoire can predict their response to PD-1 blockade. This makes sense if you think about it — all the drug is doing is opening the gates; there has to be an immune response ready to fight in the first place.

Well, it turns out that we basically invented the technology that can measure immune system clonality (amongst other things) using next-generation sequencing. Anybody else see value in a quick lab test that predicts the effectiveness of a miracle drug costing hundreds of thousands of dollars?

Me too. WOOOOOO HOOO!

* When your immune system is gearing up to fight a particular bad guy, it creates millions of copies or “clones” of the specific sequences that recognize just that antigen. This is quite different when you’re healthy — your immune system then is much more diverse, and has smaller amounts of lots of different sequences, all hunting for invaders to show up.

Advertisement

Context is king!

Our marketing materials often quote the fact that our database contains billions of unique T- and B-Cell sequences. Seriously, that is an insane number. But in isolation, it can also be misleading. After all, your body pretty much creates sequences at random in its attempt to find ones that work — and a billion rolls of a die don’t tell you very much.

The magic happens when you add context to these sequences — tracking diagnoses and outcomes, demographics, medication use, other markers, really anything that can help complete the whole picture. This stuff combined with our sequence data is what’s (really) changing the world.

In fact, one of the toughest things about good science is picking the right attributes — traditionally they’re expensive and complex to track, and lots of them don’t even matter. This is the reason we try to help our customers with Analyzer features like projects and tagging. And over the next few years, we’ll continue to add more and more features that make it easier and quicker to figure out exactly what matters most in immunology.

Zooming out, though — what if it wasn’t so hard to collect and track this context? What if we could just track “everything” about our subjects and then use statistics to automatically figure out which attributes matter. Yes, this is the promise of big data we all love to talk about, but there are many more barriers to making it real than just more and bigger computers.

Companies like Patients Like Me and 23andMe have taken a really novel approach to this challenge. What if we enlist the subjects themselves to contribute data over time, from lots of different sources? What if researchers could re-contact those subjects to ask them new questions along the way? And what if the subjects gave consent for lots of different folks to use their information in flexible and informal ways, freeing up at least a little work from the slow-moving IRB process?

The tradeoff is a fascinating one — are we better off using small bits of highly reliable and curated data, or using tons and tons that we know is noisy? Well, actually it’s not a tradeoff at all. So long as you know what you’re working with, both can be incredibly productive ways to increase our understanding of the world.

We think about this a lot, and are making real investments to help us all better understand the adaptive immune system. Thanks for joining us on this really, really fun ride.

The definition of insanity…. works!

Not everything we do here is about fancy biology; sometimes it’s about fancy web engineering. Late last week was a good example — my favorite bug since starting at Adaptive. Fair warning, this post ranks pretty high on the geek scale.

Nothing hurts my stomach more than knowing my systems are misbehaving in some way I can’t explain. I just don’t get how folks can sit by and just ignore this; it’s way too much of a threat to my ego. Screw you, Skynet — I tell YOU what to do!

Anyways, here’s the setup. Quite frequently — not enough to reproduce it in a debugger, but often enough that we were getting a steady stream of user complaints — our web servers were sending garbled responses. This manifested in a bunch of different ways. Sometimes the browser would just render a bunch of un-interpreted HTML. Other times it would screw up AJAX logic and just make the pages act wonky. It wasn’t clear at first that these were all the same things — it just felt like the site was on fire, and we had no obvious leads to work from. But we had just propped new code before this started happening, so of course that was the obvious target.

If you want to get good at debugging, especially in distributed systems, here is the #1 thing you have to remember: KEEP LOOKING. Our local hardware store has one of those big signs they put pithy statements on, and one of their favorites is “The definition of insanity is doing the same thing and expecting a different result.” At least inasmuch as it applies to debugging, this is crap.

Again and again, it’s been made clear to me that good debuggers are the ones that keep looking at the data over, and over, and over, until the patterns finally pop out. Most people peter out and say “that’s impossible” or “there’s nothing to see here” … and that is simply WRONG. The pattern is always hiding in there somewhere, and if you keep looking you will find it.

In this case, I looked at the same logs dozens of times, and followed a bunch of dead ends, before the pattern finally peeked out. Not exactly at the same time, but really close to it, we were always seeing “HEAD” requests to the server right around the calls that would fail. I ignored these for hours because they shouldn’t have made any difference. But…..

OK, here’s where things get super-nerdy. Starting way at the beginning … your web browser talks to web servers using something called “HTTP” or Hypertext Transfer Protocol. In a nutshell, the first version of HTTP worked like this:

1. The browser opens up a connection to the server computer. This is like dialing a phone and having the server answer.
2. The browser sends a message over the connection that says “I’d like your homepage, please.”
3. The server sends the HTML code that represents the site’s homepage and then hangs up the connection.

This worked great, except that step #1 was kind of slow — typically a browser will need to request not just one but many different pages and resources from the server, so “redialing” over and over was wasteful. So the protocol was updated with something called “keep-alive”, in which case the connection is kept open and used for multiple requests.

But this presented a small problem. The only way the browser knew the page was “done” was by noticing that the server had hung up the connection. If that connection stays open, how does the client figure this out? Very simply — in this new version, the server tells the browser how much data it’s going to send:

1. The browser opens up a connection to the server computer.
2. The browser asks for page #1.
3. The servers says “ok, this page is 4,000 bytes long. Here you go.” And then sends the data.
4. The browser reads out those 4,000 bytes and then using the same connection asks for page #2.
5. The server says “ok, this one is 2,000 bytes long. Here you go.” And so on.

This is way more efficient. OK, so file that one away for a moment.

Another feature of HTTP is that the browser can ask for data in a few ways. The most common is “GET”, which just asks the server to send the data for the page, thank you very much. But sometimes the browser doesn’t need the actually data for a page, it just needs to see if it’s still there and check if it’s changed since the last time it looked. For this, it can make a “HEAD” request. The HEAD request works like this:

1. The browser opens up a connection to the server computer, like normal.
2. The browser makes a “HEAD” request for page #1.
3. The server says “ok, this page is 4,000 bytes long, and it last changed on 12/1/2014.” But it doesn’t send the actual data … just general information like the size of the page.

These two concepts — “keep-alive” and “HEAD vs. GET” — were the key to this bug.

Last setup: our app is built on an open-source technology called the “Play Framework.” Play helps us match up browser requests to code, makes it easier to build pages, blah blah … not very important here. But what *is* important is that we don’t expose the Play application directly to browsers. We use a common technique called “proxying” that isolates the system a bit from the Internet. We do this with another open-source tool called the Apache web server. So our setup looks like this:

1. Browser makes an HTTP request to Apache.
2. Apache “relays” this request to Play.
3. Play responds to Apache.
4. Apache sends the response back to the browser.

The key here is that those connections between Apache and Play just use plain old HTTP. And they use keep-alives, so that many different browser requests can “reuse” the same proxy connection between Apache and Play.

Back to those HEAD requests. When a browser makes one, Apache dutifully relays it to Play. And FINALLY, here is the bug: Play was answering “ok, this page is 4,000 bytes long, and it last changed on 12/1/2014.” BUT IT WAS ALSO SENDING THE PAGE DATA, even though this was a HEAD request. This is a violation of the HTTP protocol! So after Apache read off the first part, it just stopped reading, which left all the other stuff waiting, unread, in the connection buffer.

But remember, because of keep-alive, that connection is still open. So the NEXT time that a browser asks for a page, Apache again dutifully relays it to Play over that connection, and then tries to read the response. But because it never read out the contents from the first request, all it sees is what now looks like a bunch of garbage!

From here on out things can go a bunch of different ways, depending on the specific garbage that is sent back. But it doesn’t really matter, the damage is done. Until that connection gets reset, every browser request that uses it ends up being wonked up.

And guess what? This bug has been sitting in our code since the site launched, long before I even started working at Adaptive. But it was never really exposed, because HEAD requests are generally pretty rare. As it turns out, our operations team had (ironically) just turned on a new monitoring tool that, quite legitimately, used HEAD as one of its ways to see if the site was working properly. So the bug had nothing to do with that code prop. It was classic Heisenberg.

DAMN, SON. That was a long way to go for a stupid little bug.

But there was a point, and it’s worth saying again: KEEP LOOKING. Look at the logs, again. Try running the same request, again. Look at source, again. Look at network traces, again. Look at the code, again. It is the only way to break some of these logjams. Eventually, you will pick out the pattern.

If you’re good at this — I will hire you in a millisecond. You’re gold.

We made a thing!

I’ve had a lot of jobs, but they’ve always been about building software or services — virtual stuff. Drugstore.com was fun because we shipped actual things, and visiting the warehouse was like a super-cool playground of awesome machines (pick-to-light was my absolute fav). But I’ve never actually been a part of making real, physical things for sale — until last week!

Check out our announcement of the immunoSEQ (TM!) hsTCRB kit. “hsTCRB” is apparently obvious secret code for “human t-cell receptors, beta chain,” i.e., the first of many versions of the assay that we’ll be selling in this form for research use.

This is cool because it basically explodes the volume of tests we can do. Traditionally, we’ve run a service business — folks send us physical samples (blood, tissue, etc.) and our lab deals with everything from there — DNA extraction and concentration, both amplification steps and sequencing. Only then does my team jump in, run the data through our processing pipeline and deliver results and visualizations through the immunoSEQ Analyzer.

Running a lab is a big deal — it takes equipment, sequencers, reagents, and perhaps most of all lots of people. I love our lab team and we’ll need them and more of them forever, but we simply wouldn’t be able to scale the physical processes fast and cost-effectively enough achieve our goals with an exclusively service-based business. Beyond that, lots of institutions just want to do their own chemistry, for reasons ranging from economics to privacy and environmental control.

The kit (mostly) frees us from these physical limitations and lets us scale up digitally, something I know pretty well. All those steps before the data pipeline can be done by our customers, then they use a little utility tool to send the raw sequencer output my way and we’re off to the races. Especially as we transition our pipeline up into the cloud (“on the line,” Steve!) … this gives us near infinite ability to accommodate new customers. Pretty sweet!

(You know what the limiting factor becomes? It’s the dirty secret of big data — bandwidth. Our first kit works exclusively with the Illumina MiSeq, which is a cool but mini-sequencer that generates about 1-4GB of data per run. Internally we mostly use HiSeqs, which generate 250GB of base call data alone. This stuff takes a long time to move! So much so that even on our internal networks we consider data transfer time when we’re forecasting how much we can process. Crazy.)

Anyways. Some fun issues:

• Hey, the label printed about half a centimeter askew, and now you can’t read the unique number that is the KEY TO THE WHOLE PROCESS.

• Wait, our customers aren’t all in Seattle. Does relative primer efficiency change at different ambient temperatures? Humidity? Altitude? Better do some tests ….. a LOT of tests.

• Well, this is an interesting race between customs processing time and dry ice melt time…..

• Wow, these screenshots we printed into the manual LAST YEAR don’t look right anymore…

• I’m not actually sure if our supplier has a shelf big enough for this stuff.

• Expiration date? Hmm. More tests.

And now, finally, our brand new sales team can get out there and sell the heck out of this thing. I’m thinking stocking stuffers for your whole family. Interested? Hit me up!