How would you respond to "I shouldn't have to"?

Tweet

I admit I'm scatterbrained sometimes. I forget to tell people things. I remember them when that person is not around, but by the time that person is around, my mind is off on something else. Today I took Cary to work at 4, and he told me he was off at 10:30. Tony came home some time after, but he was outside all evening working on his car. Around 8:30, he came in and asked me when Cary gets off, so I told him, 10:30. He got mad and asked me why I didn't tell him. Well, I just froze up and said, "wow". That got me an angry "wow" back and he walked out. I figured, well if he's going to treat me like there's something wrong with me for not being perfect, I'd try being a little bit more perfect, and I'd go pick up the boy myself, only when I went downstairs, he had already left. Now that I think about it, I didn't tell him because I only saw him for like 2 minutes all night, and I would have told him at some point, only he came in and asked first. In the past, when he's asked me why didn't you tell me, I've asked, why didn't you ask. His response is always "I shouldn't have to". What would you say to that?