Number of road accidents is continuously increasing in last few years worldwide. As per the survey of National Highway Traffic Safety Administrator, nearly one in five mo- tor vehicle crashes are caused by distracted driver. We at- tempt to develop an accurate and robust system for detect- ing distracted driver and warn him against it. Motivated by the performance of Convolutional Neural Networks in com- puter vision, we present a CNN based system that not only detects the distracted driver but also identifies the cause of distraction. VGG-16 architecture is modified for this par- ticular task and various regularization techniques are im- plied in order to improve the performance. Experimental results show that our system outperforms earlier methods in literature achieving an accuracy of 96.31% and processes 42 images per second on GPU. We also study the effect of dropout, L2 regularization and batch normalisation on the performance of the system. Next, we present a modified version of our architecture that achieves 95.54% classifica- tion accuracy with the number of parameters reduced from 140M in original VGG-16 to 15M only.