Asian Journal of Probability and Statistics

Hidden Markov models are powerful tools used in studying various dynamic systems, especially where internal transitions are not directly observable. In statistical manpower planning, these models help capture the effects of hidden heterogeneity on personnel transitions across states of the system. However, in practice, a significant challenge is how to determine the number of hidden states within the model from data, as this choice is often made subjectively because observable data on hidden states are nonexistent. This study presents systematic approaches to handling this problem, in a general departmentalized framework, through two search procedures. The proposed search procedures are formulated to allow any suitable statistical tools. Specifically, Likelihood Ratio statistic, Akaike Information Criterion and Bayesian Information Criterion were applied through the procedures to identify the most suitable number of hidden states for three Hidden Markov manpower model applications reviewed. All the procedures with the various statistical tools employed gave the same and consistent results. In particular, the optimum number of hidden states was found to be two for all the manpower datasets analyzed, raising curiosity about the existence of general threshold points beyond which the addition of more hidden states in hidden Markov manpower models has no significant gain. The procedures are adaptable to other application areas, outside manpower systems, where application of hidden Markov models poses the same problem of choice of number of hidden states.