Patterson played in eight games in 1975 as a freshman and made 35 total tackles. In 1976, he became the top right defensive tackle. He started eight games in 1976 and had 39 total tackles that year. In 1977, Patterson was listed as the number two defensive tackle behind Allen Pisula. He collected 11 tackles in three games. In 1978, Patterson was still the number two defensive tackle behind Allen Pisula. He missed the remainder of the 1977 season when he suffered a severe knee injury in the Virginia game. Patterson was an All-State defensive tackle at Brownsville High School for Coach Jack Henck. He lettered in football, basketball and track. Patterson majored in Physical Education.