Sam is an Uber driver who owns a 13-year old Toyota Camry. When the car is in working condition, he is able to make $400 per day. When it has been operating for i days since maintenance, the probability it fails in the current day is 1 − q −i and the probability it does not fail is q −i . If it fails at any time during the day, the cost of repairing it is $1000, and it will take that day and the next day to repair it. Assume that if it fails during a day, no revenue is generated during the day it fails or the next day. After repairs, the car will be as good as new. On the other hand, Sam can schedule preventive maintenance at the beginning of the day at a cost of $200. Assume that preventive maintenance takes the whole day (and hence no revenue is generated during that day and no extra days are needed as is the case with a car that fails), yet the car is considered as good as new (i = 0) at the beginning of the next day. No preventive maintenance can be done if the car fails or if it is being repaired. Assume that Sam is required to pay for repairs on the day it fails, and no additional costs are incurred while the car is being fixed. Formulate the maintenance problem for the next 4 days as an MDP, clearly stating the decision epochs, states, actions, transition probabilities and rewards. Assume that, regardless of the age of the car, no terminal reward is received at day 5. In addition, assume that the car has just been repaired (0 days old) at the beginning of the first day.
