James graduated from Deshler High School and Howard College in Birmingham. After completing college he helped to start the Calhoun Community College baseball program and served as pitching coach while teaching business courses at the school.As an athlete at Deshler he was All-American in football. Red received the Grogan Trophy as the top player in the area. He also lettered 5 years in basketball and was an outstanding pitcher/outfielder on the baseball team. For a brief time he played on the Robbins Floor Products Corporate Team and was drafted by the New York Giants. He went to spring training with them and was assigned to play for Michigan City, a minor league team in Indiana. He chose instead to sign a football scholarship with Howard College now Samford where he was a three sport athlete playing basketball, baseball and running track for the Bulldogs. James pitched the first no hitter in the history of the college at that time.