Concerns about Toxoplasmosis

We have one cat, who deals with the mouse problem in the house. I don't know if he has ever caught one, but the mice seem to bugger off when he's around.

DS is off to college in the fall. He's been handling the litter box, but when he goes, it's going to be my job. I could get DH to do it, but - golly - that would be a marriage strain, trust me. Plus, DH spends up to half the month traveling for his job.

If I wear gloves and an n-95 mask, am I good to go on this task?

I've asked the nurses, and one nurse thought it was OK, but the other one said it was a problem for me to do it as I am usually slightly neutropenic (sp?). There doesn't seem to be clear information on the risk.

I have to keep the cat, or there's mouse poo everywhere. It's the downside of living 20 feet away from the woods. Plus, I like His Royal Furriness. He's not a velcro kitty, but he does consent to petting of his non-belly regions during TV time at night. He's also fond of administering justice to the dog when she gets a little too keen on licking him.