Your intuition is sound, but it has to be based on some prior expectation about the balls in the bag.
For example, if you thought before you drew the blue ball that the bag was equally likely to contain any number of blue balls from zero to 10, after pulling it out you think that the probability of k blue balls remaining in the bag is (k + 1) / 55 for k = 0 to 9. That means the expected number of blue balls is now 6. Before you drew the ball the expected number was 5, so even though there is one fewer blue ball in the bag, you expect there to be one more.
But suppose instead that before you drew the first ball you thought there was a 50/50 chance of zero blue balls and one blue ball. After the draw you know there are no blue balls left, so your expectation went down from 1/2 to zero