The Shrimpers paid the price for Bilel Mohsni's first-half red card at Boundary Park as the home side went on to dominate with a composed display.
Oldham goalkeeper Alex Cisak had to make two fingertip saves in the first 18 minutes, denying both Neil Harris and Mohsni from distance.
Latics defender Zander Diamond went close, but the League Two visitors threatened again as Cisak's agility kept out Harris.
Southend had Mohsni sent off in the 38th minute, the Frenchman dismissed for two yellow cards earned by late challenges on Kieran Lee and Dean Furman in the space of 15 minutes.
Oldham took control and grabbed a 58th-minute breakthrough when James Wesolowski's lay-off set up Taylor to smash home from 10 yards.
Diamond hit a post as Paul Dickov's side chased a second, but Southend's 10 men created little in return.
If you have something to say on the present exciting times at Southend United or an idea for a new poll, then click on the link with your article or contact details and we will get in touch, just a few lines will do: www.southendunited-mad.co.uk