Avoid One Thing was started as a side project by The Mighty Mighty Bosstones' bassist Joe Gittleman. The Boston-based group's first album, titled Avoid One Thing, was released in 2002 on Side One Dummy Records. The album was written and recorded almost solely by Gittleman. Gittleman assembled a band with which to tour and play his music. Before the release of the band's next album in 2004, the band went through several lineup changes.