McCoy played music under a variety of stage names but is best known as "Kansas Joe McCoy". Born in Raymond, Mississippi, he was the older brother of the blues accompanist Papa Charlie McCoy. As a young man, McCoy was drawn to the music scene in Memphis, Tennessee where he played guitar and sang vocals during the 1920s. He teamed up with future wife Lizzie Douglas, a guitarist better known as Memphis Minnie, and their 1929 recording of the song "Bumble Bee" on the Columbia Records label was a hit. In 1930, the couple moved to Chicago where they were an important part of the burgeoning blues scene. Following their divorce, McCoy teamed up with his brother to form a band known as the Harlem Hamfats that performed and recorded during the second half of the 1930s.
In 1936, the Harlem Hamfats released a record with the song "The Weed Smoker's Dream" on it. McCoy later refined the tune, changed the lyrics and titled the new song "Why Don't You Do Right?" for Lil Green, who recorded it in 1941. It was covered a year later by bothBenny Goodman and Peggy Lee, becoming Lee's first hit single. "Why Don't You Do Right?" remains a jazz standard and is McCoy's most enduring composition.
At the outbreak of World War II Charlie McCoy entered the military, but a heart condition kept Joe McCoy from service. Out on his own, he created a band known as 'Big Joe and his Rhythm' that performed together throughout most of the 1940s. The band featured Robert Nighthawk on harmonica and Charlie on mandolin. In 1950, at the age of 44, McCoy died of heart disease in Chicago, only a few months before Charlie. They are both buried in Restvale Cemetery in Alsip, Illinois.
McCoy also performed and recorded under the names Bill Wither, Georgia Pine Boy, Hallelujah Joe, Big Joe McCoy and His Washboard Band, and The Mississippi Mudder. He also used the names Hamfoot Ham, Hillbilly Plowboy, and Mud Dauber Joe.
Like many blues musicians of his era, Joe McCoy's grave site was originally unmarked. A tribute concert took place in October 2010 to celebrate the music of Joe and Charlie and to buy gravestones for each of them; they were installed on May 31, 2011.