Olsen's racing career spanned 25 years before hanging up his helmet in 1973. Olsen captured the 1951 NARA Sprint Car championship and in 1958 he was the NASCAR National Modified champion as well as the winner of the Garden State Classic at Wall Stadium. Olsen won the first ever race at Trenton Speedway in 1958 on asphalt.

In 1961, at one of his starts at Langhorne Speedway, Olsen charged to the front and at the checkers, he narrowly missed winning the race. Olsen placed second in a photo finish with Dutch Hoag.

In 1964, Olsen won the Reading Speedway Modified championship.

He retired in 1973 started building race cars. Between 1973 and 1979, Olsen built more than 200 cars.