Southeast Asia supports high biodiversity, in a mosaic of forest types formed by the expansion and contraction of habitats through past climate changes. Among the region’s forest types, the geographical distribution of peat swamp forests has fluctuated intensely over the past 120,000 years. Most peat swamp forests in Southeast Asia are found in coastal regions and formed within the last 7,000 years after a decline in sea level. However, some peat swamps were initiated earlier on substrates of slightly higher elevation, and these peat swamps might have been refugia for peat swamp species in the last glacial period and the high sea level period. We assessed genetic diversity, genetic structure and divergence time of current genetic groups for Shorea albida in Brunei, an endemic tree species of Bornean peat swamp forests, using 18 microsatellite markers. Genetic diversity was not lower than has been found in other Shorea species, possibly because of the high density of S. albida in Brunei. Although overall genetic divergence between populations was low, two populations (Ingei and Labi Road 3) were distinct from the other populations. Analysis using DIYABC estimated that three genetic groups (Ingei, Labi Road 3 and others) diverged simultaneously from their ancestral population, whose effective size was very small, about 7,500 years ago, corresponding to a recent sea level peak in the Belait-Baram river basin. In that high sea level period, some higher-elevation lands remained, and peat formation had already started in this region. We propose that the current genetic structure of S. albida in Brunei was formed from small refugial populations that survived the period of higher sea level in these higher-elevation areas. Because of their relatively high genetic diversity, Brunei’s S. albida populations should become an important genetic resource for the recovery of genetically healthy populations in other parts of northwest Borneo.