Problem 1: (18 points) The Markov chain with transition probabilities listed below is in state 3 immediately before the first trial. p1,1 = p2,2 = 0.4, p1,2 = p2,1 = 0.6, p3,2 = 0.5, p3,3 = 0.2, p3,4 = 0.3, p4,5 = p5,6 = p6,4 = 1.0.1. Draw the state-transition diagram for this Markov chain. Indicate which states, if any, are recurrent, transient, and periodic.2. Find the probability that the process is in state 3 after n trials.3. Find the expected number of trials up to and including the trial on which the process leaves state 3.4. Find the probability that the process never enters state 1.5. Find the probability that the process is in state 4 after 10 trials.6. Given that the process is in state 4 after 10 trials, find the probability that the process was in state 4 after the first trial.
