The brick wall tubes are popular forms of ancient masonry pagodas which can be seen as a spatial lateral force resist system. The masonry of the ancient pagoda is the case of compression and shear affected by earthquake, and the composite compression and shear behavior is one of the key issues to the seismic capacity of masonry tube structure. In order to study the mechanical properties of sub-structure of masonry pagodas, taking the Xuanzang Pagoda as the prototype structure which lies in the Xingjiao Temple in Xi'an City, three sub-structure models were designed and constructed. The low cyclic loadings tests were conducted to the models and the crack, deformation and failure phenomena are surveyed during the loading process. Then the simulation models are developed for calculation, the results are acquired about the equivalent strain, load-displacement curve. Comparing the calculated results with the experimental results, the effects of vertical compressive stress on the failure characteristics, hysteretic performance, ductility and energy dissipation capacity of the masonry in the ancient tower were analyzed. As results, the error was less than 21% for the calculated value of the characteristic load relative to the test value. The distribution of equivalent plastic strain was consistent with the crack failure area of specimens. When the vertical pressure remains constant, with the increase of horizontal load, the tower body gradually cracks and damages along the masonry joints, and the width of cracks also increased. The failures around the structure opening were more obvious, and the residual deformation of specimens increases. When the vertical pressure was increased, the cracking and residual deformation of specimens were limited, but the failure process characteristics of specimens had little changes. With the increase of the ratio of compression to shear, the range of cracking and damage of the masonry of the ancient tower decreased, and the shear bearing capacity, the stiffness and energy dissipation capacity were increased, but the ductility and deformation capacity were slightly decreased. The results can provide references for the assessment to structural damage and seismic capability of ancient masonry pagodas.