Allen faced personal tragedy in his later years with the death of his first born, Ivan Allen III, in 1992, and it was difficult for the former mayor to recover from this terrible event. His health declined. He had kept somewhat active with the Ivan Allen Company, remaining chairman until 1995, but his sons had gradually taken over business matters, first Ivan Allen III, and then H. Inman Allen.
Allen’s later years still found him interested in city matters, although removed from them. He remained a city booster and was overjoyed when Atlanta secured the Olympics for 1996. City leaders understood and appreciated his years of service and the importance he played in dealing with the civil rights struggles. Maynard Jackson spoke warmly of Allen at a testimonial dinner in 1993, praising the former mayor for forging improved race relations in the city. Allen, he said, made Atlanta a better place to live. The former mayor received appreciation for his important role in many other ways, including the naming of Georgia Tech’s Liberal Arts College as the Ivan Allen College of Liberal Arts in 1990.
Ivan Allen, Jr. died in July 2003 at the age of 92.