For wireless communication in the high-frequency band, Intelligent Reflecting Surface (IRS) has been developed to expand the coverage. To appropriately control the reflection pattern of each element in the IRS, deep learning (DL)-based spectral efficiency predictions have been proposed. The conventional method performs prediction from the partially estimated channel at a single point in time. However, since the movement of wireless terminals is spatially continuous, the accuracy can be improved using past estimated channels. Therefore, in this paper, we propose a prediction method that considers the movement of wireless terminals by treating the estimated channel as time-series data. Furthermore, we apply a long short-term memory network to capture the time-series nature efficiently. The numerical experiments show that the proposed method can achieve high spectral efficiency even with smaller training samples than the conventional method.