Plotting Markov Chains in R

General
1

I am working with the R programming language.

I thought of the following "game":

  • There are two coins : Coin 1 and Coin 2
  • The player picks a random coin (i.e. 0.5 probability of picking either Coin 1 or Coin 2)
  • Coin 1 has a 0.5 probability of landing on Heads and a 0.5 probability of landing on Tails
  • Coin 2 has a 0.3 probability of landing on Heads and a 0.7 probability of landing on Tails
  • If Coin 1 is Heads, the player gets a score of 1 else 0
  • If coin 2 is Heads, the player gets a score of 2 else -1
  • The player repeats this 3 times and records his final score
  • Note: The player starts the game with score = 0

I want to make a "Markov Chain" (e.g. Getting Started with Markov Chains | R-bloggers , Markov chain - Wikipedia, Markov Chains and decimal points in r?) for this game that shows the probability of what score the player can have conditional on the current turn and current score - based on "simulated data" from this game. I wrote a simulation that plays this game 100 times:

library(dplyr)

results <- list()

 for (i in 1:100) {

iteration = i

coin_turn_1_i = sample(sample(c(1,2)), size =1, replace = T, prob= c(0.5,0.5))

score_turn_1_i = ifelse(coin_turn_1_i == 1, sample(sample(c(1,0)), size =1, replace = T, prob= c(0.5,0.5)), sample(sample(c(2,-1)), size =1, replace = T, prob= c(0.3,0.7)) )

coin_turn_2_i = sample(sample(c(1,2)), size =1, replace = T, prob= c(0.5,0.5))

score_turn_2_i = ifelse(coin_turn_2_i == 1, sample(sample(c(1,0)), size =1, replace = T, prob= c(0.5,0.5)), sample(sample(c(2,-1)), size =1, replace = T, prob= c(0.3,0.7)) )

cum_score_2_i = score_turn_2_i + score_turn_1_i

coin_turn_3_i = sample(sample(c(1,2)), size =1, replace = T, prob= c(0.5,0.5))

score_turn_3_i = ifelse(coin_turn_3_i == 1, sample(sample(c(1,0)), size =1, replace = T, prob= c(0.5,0.5)), sample(sample(c(2,-1)), size =1, replace = T, prob= c(0.3,0.7)) )

total_score_i = score_turn_3_i + cum_score_2_i


my_data_i = data.frame(iteration, coin_turn_1_i, score_turn_1_i, coin_turn_2_i, score_turn_2_i, cum_score_2_i, coin_turn_3_i, score_turn_3_i, total_score_i )

results[[i]] <- my_data_i

}

results_df <- data.frame(do.call(rbind.data.frame, results))

I tried to (manually) extract the "transition probabilities" between "turn 1 and turn 2" and "turn 2 and turn 3":

#turn 1 and turn 2

turn_1_and_turn_2 =  data.frame(results_df %>% group_by(score_turn_1_i, cum_score_2_i) %>% summarise(count = n()))

results_df %>% group_by(score_turn_1_i) %>% summarise(count = n())
# A tibble: 4 x 2
  score_turn_1_i count
           <dbl> <int>
1             -1    25
2              0    22
3              1    28
4              2    25

# transition probabilities 
turn_1_and_turn_2$prob = ifelse(turn_1_and_turn_2$score_turn_1_i == -1, turn_1_and_turn_2$count/25, ifelse(turn_1_and_turn_2$score_turn_1_i == 0, turn_1_and_turn_2$count/22, ifelse(turn_1_and_turn_2$score_turn_1_i == 1, turn_1_and_turn_2$count/28, turn_1_and_turn_2$count/25)))

# turn 2 and turn 3


turn_2_and_turn_3 =  data.frame(results_df %>% group_by( cum_score_2_i, total_score_i) %>% summarise(count = n()))

results_df %>% group_by(cum_score_2_i) %>% summarise(count = n())

# A tibble: 7 x 2
  cum_score_2_i count
          <dbl> <int>
1            -2     5
2            -1    13
3             0    22
4             1    20
5             2    21
6             3    13
7             4     6

turn_2_and_turn_3$prob = ifelse(turn_2_and_turn_3$cum_score_2_i == -2, turn_2_and_turn_3$count/5, ifelse(turn_2_and_turn_3$cum_score_2_i == -1, turn_2_and_turn_3$count/13, ifelse(turn_2_and_turn_3$cum_score_2_i == 0, turn_2_and_turn_3$count/22, ifelse(turn_2_and_turn_3$cum_score_2_i == 1, turn_2_and_turn_3$count/20, ifelse(turn_2_and_turn_3$cum_score_2_i == 2, turn_2_and_turn_3$count/21, ifelse(turn_2_and_turn_3$cum_score_2_i == 3, turn_2_and_turn_3$count/13, turn_2_and_turn_3$count/6))))))

My Question: But from here, I do not know how to take "turn_1_and_turn_2$prob" and "turn_2_and_turn_3$prob" and convert them into a transition matrix - and then convert them into a "Markov Chain" :

enter image description here

Of course - I could manually enter the all the above probabilities into a matrix object:

#for a n-state transition matrix
transition_matrix = matrix(1:n^2, nrow = n, ncol = n)

But can someone please show me how to make this kind of Markov Chain directly from "results_df"? Is there a general method that can be used for "results_df" if the player flips the coin as much as he wants (e.g. 4 times)?

Thanks!

Note: In this game I have created, it seems that the game can end in 9 possible states - so I think this means that the transition matrix would be 9x9?

turn_2_and_turn_3 %>% group_by(total_score_i) %>% summarise(count = n())
# A tibble: 9 x 2
  total_score_i count
          <dbl> <int>
1            -3     1
2            -2     2
3            -1     3
4             0     3
5             1     4
6             2     4
7             3     4
8             4     3
9             5     2
2

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.