Part 1

So you want to understand how much you can trust test results? I've got you covered.

Warning: There's enough math here to make your eyes cross and induce high school flashbacks. But I’m from Missouri, the Show Me state, so this is for all the people who were muttering, "Oh yeah, prove it!" while reading part 1 (see link above top line). There's no calculus or chaos theory here; just some basic addition/subtraction and multiplication/division. You'll survive — promise!

Let's look at some examples with actual numbers.

Example 1

Your elderly neighbor died and you've adopted her cat, Petunia. She seems healthy but you don't have any of the cat's medical records. Your veterinarian offers to test her for feline leukemia (FeLV). Should you do it?

We need three numbers to help make that decision. The prevalence of the disease; the sensitivity of the test; and its specificity. Prevalence is the proportion of animals that have the disease out of an entire population of cats. Since evaluating every single cat isn't a viable option, we evaluate a sample population and extrapolate from there. The sensitivity of a test is a measure of how well it does at flagging all individuals that have the disease. The specificity is a measure of how well it avoids flagging individuals that don't have the disease. In an ideal world, the sensitivity and specificity would both be 100 percent, but this isn't an ideal world so we're usually pretty happy with anything more than 95 percent.

Let's say the prevalence of FeLV in cats in the U.S. is about 3 percent. The FeLV test offered to you has a 97 percent sensitivity (Se) and a 98 percent specificity (Sp). We'll use a table to keep track of the numbers.

Our prevalence is 3 percent, so out of any 100,000 cats, 3000 will have feline leukemia.

Our sensitivity of 97 percent means that the test will correctly flag 2910 as diseased (0.97 x 3000). These are called true positives. The other 90 are diseased but the test didn't flag them — these are the false negatives.

A little basic addition lets us fill in the numbers of cats we expect to test positive or negative whether or not they actually have the disease.

So who cares about the table? Does it really tell you anything about *your* cat? Nope, not really. But this table lets us calculate predictive values. Those values are going to tell us how to interpret a positive test and a negative test.

The positive predictive value (PVP) tells you how likely a cat that tests positive is to actually have FeLV. It's calculated by dividing the true positives by all positives (2910/4850). We come up with 0.6, which means a cat that tests positive for FeLV has a 60 percent chance of actually being infected with FeLV. That's better than a coin toss, but not by much. If the cat you inherited tests positive, you'll want to consider confirming it with a different test.

The negative predictive value (PVN) tells you how likely a cat that tests negative is to actually be disease-free. It's calculated by dividing the true negatives by all negatives (95,060/95,150). We come up with 0.999, which means a cat that tests negative is 99.9 percent likely to be free of FeLV. That's a nice strong number so you can really trust this negative test.

This math works for “all cats in the U.S.” — if we didn’t know anything about the individual situation. But we know that Petunia was the only cat in the house and lived completely indoors. How would that affect our test interpretation? We could estimate a “prevalence” that was even lower than the 3 percent average – maybe 0.3% or 0.003 (a really, really, really, small chance of having FeLV). Let’s do the math again.

Now we calculate the positive and negative predictive values – the probabilities that if Petunia has a positive test that it’s a “true positive” and if she has a negative test, that it’s a “true negative”. The probability that a positive is a true positive comes out at about 13%. The probability that a negative is a true negative is almost 100%. So, for every 8 cats like Petunia that we test that have a positive test result, only 1 of them will be a true positive with a FeLV infection. We don’t know which one, but most of them will not actually have the disease we’re looking for. And if we tested 100,000 cats like Petunia, we could almost guarantee that each cat would test negative. With that much certainty, do I need to test Petunia for feline leukemia virus?

Example 2

But what if you find out that your inherited cat wasn't properly vaccinated and frequently went outside? A couple weeks after you adopt her, you notice she's been losing weight and vomiting sometimes. When you take her in to your veterinarian, she's found to have a fever too. Does that change how we'd interpret FeLV test results?

Your veterinarian keeps up to date and just attended a seminar where she learned that the prevalence of FeLV in outdoor, unvaccinated cats that are showing symptoms consistent with FeLV disease is closer to 80 percent. We'll use the same test with a 97 percent sensitivity and a 98 percent specificity, but this time the numbers look quite different.

The PVP is now 77,600/78,000 = 0.9949. That's a 99.5 percent chance that this sick cat actually has FeLV if she tests positive.

The PVN is 19,600/22,000 = 0.8909. That's an 89 percent chance that this cat doesn't have FeLV if she tests negative.

This case illustrates how important it is to interpret test results in light of the whole clinical picture. The more information you have, the better decisions you can make. A single test may not tell you much of anything.

Example 3

Now let's say you just finished reading a Facebook post about someone's dog who died of heartworm disease. OMG, could your dog be infected with a mass of worms squatting in his heart, fluttering with every single beat? He seems fine but should you rush him in to be tested anyway? Cuz OMG – MASS of actual WORMS!!!!

The prevalence of heartworm disease depends on where your dog lives and any areas they travel to. (Hint: If you see a lot of mosquitoes around, you're likely in a hot spot.) Let's say you live in Nevada where mosquitoes are pretty rare. According to the [CAPC], 13,147 heartworm tests have been reported so far this year with 56 of them positive (0.4%). [https://www.capcvet.org/maps/#2017/all/heartworm-canine/dog/united-states/nevada/] If you've been paying attention, you know there's going to be some number of false positives in there so let's round that down and say we've really got a prevalence of 0.3 percent (300 of 100,000 dogs).

We can see we're really looking for a needle in haystack here.

A common test for heartworm disease in dogs has a sensitivity of 84 percent and a specificity of 97 percent. Wow. That's not really great right off the bat, but let's see how it works out.

Our PVP is 252/3,243 = 0.077, so a dog that tests positive has a 7.7 percent chance of actually having heartworm disease.

Our PVN is 96,709/96,757 = 0.999. That's a 99.9 percent chance that your dog doesn't have heartworm disease if the test is negative.

So is it worth testing your happy healthy dog just because you read a scary article? Probably not, unless you're really susceptible to fearmongering headlines and you've got an extra $100 burning a hole in your pocket. If the test is negative, you can sleep comfortably at night confident that your dog doesn't have heartworm disease. If the test turns up positive, there’s still a teensy tiny itty bitty chance of her actually having a throbbing lump of worms in her chest but now you get to worry about it.

Example 4

But wait! What if you live in Missouri where heartworm disease is much more common? That changes everything, right? Nope.

The CAPC data for Missouri shows 1,308 positives out of 82,205 tests or 1.6 percent. Again, we'll drop that a bit to guestimate our real prevalence, so let's say Missouri's prevalence is 1.5 percent (five times higher than Nevada).

The PVP is 1260/3230 = 0.39 and the PVN is 96,530/96,770 = 0.9975. So still less than a 50:50 chance that a positive test indicates heartworm disease, but at least you can still be confident about a negative test.

Example 5

Isn't there any screening test that you can just trust? All this 'interpret it in light of clinical signs' and 'confirmation may be necessary' stuff is too complicated.

Let's say you found a really good test with a 98 percent sensitivity and a 98 percent specificity. Surely that's good enough for blind trust. Nope.

Assume you have a 10 percent prevalence. That mean one out of 10 animals has the disease.

Our PVP is 9800/11,600 = 0.8448 and our PVN is 88,200/88,400 = 0.9977. That's pretty good — almost an 85 percent chance that a positive test means there's actually disease, and over a 99 percent chance that a negative test means there's no disease. But are you comfortable saying that you'd start chemotherapy or amputate a leg if a single blood test showed your totally normal-looking, active dog probably had cancer, but there's a 15 percent chance the result is bogus?

What if we had the kind of test doctors only dream about? Somebody finds a test with 99 percent sensitivity and specificity. Assuming we still have that 10 percent prevalence….

Our PVP is 9900/10,800 = 0.9167 and our PVN is 89,100/89,200 = 0.9989. The super-duper bestest ever test still only gives us a 92 percent level of trust on a positive result. How disappointing.

Let's go back to our excellent but still realistic test with a 98 percent sensitivity and specificity. How common would the disease have to be to have a high PVP? Try a prevalence of 40 percent.

That's a PVP of 39,200/40,400 = 0.9703 and a PVN of 58,800/59,600 = 0.9866. Finally! A test you could trust!

But a prevalence of 40 percent is extremely high. To put it in perspective, that's about the rate of obesity in American adults and less than the rate of high blood pressure. Most diseases are much less common. Diabetes is a 'common' disease in multiple species — around 10 percent of Americans have diabetes, and it's diagnosed in less than 1 percent of cats -- but that's not going to give us 'blindly trustworthy' test results.

Are you still muttering "prove it”? I applaud your tenacity! Here are some additional resources you may find useful.

Positive and Negative Predictive Value

MedCalc: Bayesian Analysis Model

Molecular Diagnostic Testing: Pros and Cons