Читать книгу Simulation and Analysis of Mathematical Methods in Real-Time Engineering Applications - Группа авторов - Страница 40

Working of Markov Chain Model - To statistically model random processes, the Markov chain model is one of a simple way. Markov Chain model is also defined as “A stochastic process containing random variables, transitioning from one state to another depending on certain assumptions and definite probabilistic rules” [19]. They are widely used in applications from text generation to financial modeling and auto-completion applications.

Using Markov property, random variables transit from one state to another. Markov property states that the probability of changing from current to the possible future state by random process depends on the current state and time and is independent of the series of states that preceded it.

Figure 2.6 illustrates the Stochastic process, which is further classified into Markov Chain, Semi Markov, Markov Decision, Hidden Markov and Queuing model.

Markov chain describes a sequence of possible events where each event’s probability depends on the previous state event [20]. Semi-Markov model is an actual stochastic model that evolves with time by defining the state at every given time [21]. As the name itself implies for Markov Decision, it is a process of making decisions where partly random decisions are few and few are at the decision maker’s control [22]. In the Hidden Markov model, rather than being observables, the states are hidden [23]. The Queuing model helps predict the length of the queue and waiting time is predicted [23].

The Markov chain decision process is a mathematical model that evaluates the system during the transition from one state to another to the probability rules. It can be classified into two types. They are discrete-time Markov decision chains and continuous-time Markov decision chains. Additionally, it can be classified based on the number of states taken to decide the next state. It will be defined as the first-order, second-order, and higher-order chains. There are many offloading schemes based on the Markov decision process [22, 23].