Your best bet is in and around Tan Dinh Market on Hai Ba Trung Street in District 3. You will find virtually any type of fabric imaginable (cotton, jersey, linen, silk, denim, etc). The stalls inside the main entrance of the market (in that center aisle) tend to be pricier than those on the side aisles, so make sure to explore once you're in there. There are also a few shops on the main street (Hai Ba Trung) that have pretty reasonable prices (Nhat Ai is one). Most of the shops will speak a bit of English.