The following two ways can be used to find the optimal value of mtry :
1. Apply a similar procedure as in finding the optimal n_tree such that random forest is run 10 times. The optimal number of predictors selected for split is selected for which the out-of-bag error rate stabilizes and reaches the minimum.
2. In this method, we are doing the experiment by including the values such as the square root of the total number of all predictors, half of this square root value, and twice of the square root value, etc and at the same time check which value of mtry gives the maximum area under the curve.
For Example, Suppose we have 1000 predictors, then the number of predictors to select for each node would be 16, 32, and 64.