Abstract: The modern rolling industry has improved product quality, and the technical requirements of high accuracy and high dynamic performance have made the issue of rolling mill vibration more prominent. Rolling mill system instability seriously affects the quality of the product, reduces the accuracy of the product, and even causes serious damage to the rolling mill equipment. During hot rolling process, friction is of great importance to vibration and stability of the rolling mill. There is a difference in the friction coefficient between the upper rolling interface and lower rolling interface. Considering the asymmetric friction coefficient, a chatter model was established by combing the rolling process model and the vertical−horizontal−torsional structure model of a hot rolling mill to study the relationship between friction coefficient asymmetry and stability of the rolling mill system. According to the mathematical model, the friction coefficient stability domain of a rolling mill system is determined by the application of stability criterion. And it shows that the influence of the asymmetric friction coefficient on the stability domain is significant. Due to the different degrees of asymmetry, the system is divided into stable domain, horizontal instability domain, and horizontal−torsional instability domain. As the asymmetry in terms of the friction coefficient becomes considerable, it would occur various vibration modes. Through a field test of a hot rolling mill, the vibration signal of the rolling mill system was obtained, which verified the correctness and validity of the simulation analysis results. The degree of asymmetry in the friction coefficient is the same when rolling the container plate and the Q235 plate, but the deformation resistance of the system is different. The system falls into the horizontal instability domain when the container plate is rolled, displaying clearly horizontal vibration. However, the system falls into the stable domain when the Q235 plate is rolled, and the system shows no obvious vibration.