Downlink scheduling in LTE is an open problem for which several heuristic solutions exist. Recently, there has been an increase in interest in applying machine learning to networking problems, including downlink scheduling. Improvements in Physical Layer capabilities have generated new resource-intensive use cases and continuously modifying existing heuristic solutions could result in the development of systems too complex to maintain. We propose a LSTM/Pointer Network-based downlink scheduler which aims to improve upon the current models which utilize feed forward neural networks. Our scheduler flexibly handles changing numbers of UEs via a recurrent neural network. We integrate the channel quality indicator and the buffer size of each UE as the observation of a MDP and solve it using a Deep Reinforcement Learning algorithm. Our experiments demonstrate that our approach results in a scheduler which generalised across changing number of UEs and resource blocks and performed within the range of traditional schedulers.