If Cape Town has a single origin monument, this is it. A pentagonal bastion fort whose first stone was laid on 2 January 1666, the Castle of Good Hope is the oldest surviving colonial building in South Africa, and one of the best-preserved seventeenth-century Dutch East India Company forts anywhere in the world. Its five bastions – Leerdam, Buuren, Catzenellenbogen, Nassau, and Oranje – were named after titles held by Willem III, Prince of Orange, and those names now appear on streets across Cape Town's suburbs.

The building material itself is part of the story: stone quarried from Robben Island and slopes of Table Mountain, worked by enslaved labour. The yellow ochre paintwork was originally chosen not for decoration but because it reflected heat and sun in the Cape's harsh summer. The entrance bell, cast in Amsterdam in 1697 by the East-Frisian bellmaker Claude Fremy and weighing a little over 300 kilograms, is the oldest bell in South Africa and could once be heard ten kilometres away.

The Castle still functions as the regional headquarters of the South African National Defence Force's Cape Regiments, which makes it one of the very few UNESCO-class heritage buildings worldwide that is simultaneously an active military installation. Visitors can watch the Key Ceremony, a ritual based on actual seventeenth-century drills, Monday to Friday at 09:00 and 14:00. Inside the walls are three museums – the Military Museum, the William Fehr Collection of colonial-era painting and decorative arts, and the Iziko Cape Heritage Museum – plus the ornate Kat Balcony, designed by Louis Michel Thibault with reliefs by sculptor Anton Anreith in the 1780s.

Walk up Adderley Street, cross Wale Street, and the tarmac gives way to a corridor of oaks. This is the Company's Garden – the place Cape Town was founded for. Established by the VOC in 1652 as a vegetable plot to resupply ships on the long sea route to Batavia, it is by a considerable margin the oldest continuously cultivated public space in South Africa, and one of the oldest in the southern hemisphere.

The Fresh River that once watered Khoekhoe cattle and drew San hunters before them still runs through stone canals beside the main avenue. You can, on a quiet morning, hear it under the grates. What began as rows of cabbages and turnips for scurvy-stricken sailors is now six hectares of formal lawns, rose gardens, an aviary, a Japanese koi pond, and – as every local will eventually tell you – squirrels so habituated they will climb onto your shoulder.

The Garden is also an institutional district. Ringing its lawns you find Parliament, the South African Museum (the country's oldest, founded 1825), the South African National Gallery, the Iziko Planetarium, the National Library, Tuynhuys (the official Cape Town residence of the President), and the Bertram House Museum. Few cities in the world let you stroll in twelve minutes from a koi pond planted by Dutch sailors to a working head-of-state's front door.

On the corner where Adderley Street meets Wale, inside what looks from the outside like a pleasant, mustard-coloured Georgian building, is one of the most difficult rooms in Cape Town. The Slave Lodge was built by the Dutch East India Company in 1679 to house the human beings it trafficked to the Cape from East Africa, Madagascar, India, and the Indonesian archipelago. It is the second-oldest colonial building in the city after the Castle, and arguably the most important confrontation with the truth of Cape Town's founding that exists anywhere in South Africa.

Over the course of the next 130 years, between 6,000 and 9,000 enslaved people passed through this one building. Conditions were lethal – the mortality rate among enslaved residents is estimated to have exceeded 20% in some years. A drainage ditch carried waste beneath the floor. After slavery was abolished in the British Empire in 1834, the Lodge was repurposed: successively as the Supreme Court, as post-office, as government offices, and eventually in 1966 as the South African Cultural History Museum – a body that, notably, told almost nothing of the building's original function.

Since 1998, the building has been the Iziko Slave Lodge – a museum of slavery, human rights, and memory. The permanent exhibit "Remembering Slavery" walks visitors through the Cape's four-century experience of enslavement, with a particular emphasis on individual lives reconstructed from VOC ledgers. The wall of names, where visitors read the first names of the enslaved people who lived in this building, is the closest thing modern Cape Town has to a counter-monument.

Directly across Adderley Street from the Slave Lodge stands the Groote Kerk – the oldest Christian congregation in South Africa, founded in 1665, and mother church of the Nederduitse Gereformeerde Kerk (Dutch Reformed Church). The bell tower that looms over Church Square dates to 1704; the current main building, designed by the German-born architect Herman Schuette, was completed in 1841 after the original church became structurally unsound.

The interior is a study in disciplined Protestant understatement – plain white walls, dark-wood pews, two galleries – interrupted by two spectacular exceptions. The first is the pulpit, carved in Burmese teak by the sculptor Anton Anreith together with the carpenter Jan Graaff, inaugurated in 1789 at a cost of £708 (roughly £130,000 / R3 million in today's money). Its base is supported by two life-size wooden lions – a species now extinct at the Cape – and it is widely held to be the finest example of colonial-era ecclesiastical sculpture in southern Africa. The second is the organ: 5,917 pipes, installed in its current configuration in 1954, making it the largest pipe organ in South Africa and, according to the congregation, housed beneath the largest unsupported domed ceiling in the southern hemisphere.

The church is free to enter on weekdays and a working congregation still holds Sunday services. For any visitor interested in either music or sculpture, the Groote Kerk rewards the fifteen-minute detour even if the architecture outside is, frankly, a bit of a hybrid.

Climb the slopes above Buitengracht and the city changes colour. The Bo-Kaap – "above the Cape" in Afrikaans – is the oldest surviving residential quarter in Cape Town and the historic heart of the Cape Malay community. Established in the 1790s as cheap-rental housing for Muslim artisans, and populated after emancipation in 1834 by freed slaves, the neighbourhood now consists of tightly-packed Cape Dutch and Georgian row-houses painted in blocks of candy pink, cobalt, saffron, and mint.

The colour is not, as often claimed, a celebration of emancipation. The current palette is twentieth-century – older photographs show the buildings painted in far more muted whites and cream. But the social fact the colour now represents – a Muslim community that has resisted displacement for two centuries – is unmistakably real. During apartheid, the Bo-Kaap was narrowly spared the forced removals that erased District Six just down the hill.

At 71 Wale Street sits the Iziko Bo-Kaap Museum – the oldest surviving house in the quarter, built between 1763 and 1768. Its rooms are furnished as a modest nineteenth-century Cape Malay home and the exhibits trace the community's religious, linguistic, and culinary history. Around the corner on Dorp Street is the Auwal Mosque, founded in 1794 by Tuan Guru (Imam Abdullah Kadi Abdus Salaam, a Qadi from Tidore exiled by the VOC) – the first and therefore oldest mosque in South Africa.

On the east edge of Greenmarket Square – the city's oldest open square, where freed slaves once held their first public markets – stands a single-storey, honey-limestone building in a style so out-of-place it stops traffic. The Old Town House, built in 1755, is the finest surviving example of Cape Rococo architecture in South Africa: asymmetrical gable, rusticated quoins, a small clock tower, and finishes that feel lifted from a provincial French hôtel-de-ville.

Over the course of its 270-year life the building has served as watch-house for the burgher militia, as the seat of the Cape Town Senate, and – until the completion of the City Hall in 1905 – as the main town hall. Since 1914 it has housed the Michaelis Collection: a gift from the mining magnate Sir Max Michaelis of 17th-century Dutch and Flemish Golden Age painting, including works by Frans Hals, Jan Steen, and a small but dense selection of Dutch genre and still-life pieces. It is, outside the Rijksmuseum, one of the more surprising concentrations of Netherlandish Golden Age art in the world.

A five-minute walk from the Castle, half-hidden behind trees on Strand Street, is one of the most quietly extraordinary pieces of domestic architecture in the city. Erf 8 on Strand Street was granted by Governor Willem Adriaan van der Stel in 1701. The current building, with its beautifully resolved Neoclassical façade – four pilasters, a central pediment, a projecting lantern above the teak-and-plaster entrance – dates from around 1793 and has been attributed, though not conclusively, to the French-born architect Louis Michel Thibault.

The architectural historian Karel Linscheid has argued that the Koopmans-de Wet façade is the only building in Cape Town whose proportions consistently use the golden ratio (1:1.618) at every scale – from the overall elevation, to the window bays, to the carved details of the entrance portal. Whether this was Thibault's deliberate design or a regional builder's intuition is debated; the effect, once you know to look for it, is a sense of inevitability you normally find only in Renaissance Italian work.

The building takes its name from Marie Koopmans-de Wet (1834–1906), a formidable Cape cultural figure who held court here with her sister Margaretha and assembled – a campaign eventually backed by both Louis and Annie Botha and by the Randlord Lionel Phillips – a collection of Chinese porcelain, Delftware, Cape silver, Cape-made furniture in stinkwood and yellowwood, and household textiles now considered the first serious ceramics and decorative-arts collection in South Africa. The house passed to the nation in 1913 and opened as a museum the following year. It is currently in dire need of maintenance – visitors should manage expectations about the lighting and the state of the garden – but the façade and the craftsmanship of the interior joinery remain unique in the city.

On the corner of Wale and Queen Victoria, facing the Company's Garden, stands the cathedral that became the symbolic heart of the anti-apartheid movement. Designed by Sir Herbert Baker in 1901, built out of Table Mountain sandstone, and taking the better part of the twentieth century to complete (the nave was only finished in 1978), St George's is the mother church of the Anglican Diocese of Cape Town and the oldest Anglican cathedral in southern Africa.

The architectural debt is to English Gothic – pointed arches, ribbed vaults, a carved rood screen – but the stone is local, hauled down from the slopes of Table Mountain and Lion's Head, and the stained-glass windows include unmistakably South African themes. The cathedral is known internationally, however, for something the architecture can only frame: the moral authority it acquired under Archbishop Desmond Tutu, who served here from 1986 until his retirement in 1996, and the role it played during the apartheid years as a refuge, a meeting-place, and a pulpit from which power was repeatedly told the truth. Its nickname – "the People's Cathedral" – dates from that era.

The cathedral is an active place of worship with daily Eucharist and regular choral evensong, and it hosts a packed programme of classical concerts. The acoustics are excellent; if you can find an evening concert during your visit, take it.

A short walk up Buitenkant Street from the District Six Museum is a building that makes a very good case for being the most beautiful surviving colonial townhouse in Cape Town. Rust en Vreugd – "Rest and Joy" – was built in 1778 as a private residence for Willem Cornelis Boers, the Fiscal (head of the VOC justice system) at the Cape. Its three-storey facade, mounted on a raised stoep and set behind a double row of oaks, represents the Cape Baroque at its most refined: a shallow pediment gable, heavy teak panelled front doors carved by Anton Anreith, and fenestration that falls into strict symmetry.

Inside, the plaster ceilings and yellowwood floors are largely as they were in the 1780s. The gardens behind the house – formal, geometric, with a gazebo and lavender hedges that may be as old as the museum era itself – are frequently cited as among the most intact period gardens in the Western Cape, though they too are currently suffering from under-maintenance. The building today houses a small selection of watercolours and prints from the William Fehr Collection, including some of the earliest visual records of Cape Town.

The tenth site on this list is not a building but an entire island, and the only one on our list with UNESCO World Heritage status (inscribed in 1999 under the cultural criteria of "outstanding universal value"). Robben Island sits 12 kilometres northwest of the V&A Waterfront in the cold Atlantic waters of Table Bay, and has been used as a place of exile, isolation, and punishment almost continuously since Portuguese sailors first used it as a landing in 1498.

Its layered history includes periods as a Dutch-era prison, a leper colony, an asylum, and a Second World War military installation. But what draws almost half a million visitors a year is the island's role from 1961 to 1991 as the maximum-security prison for political opponents of the apartheid regime – a list that includes Nelson Mandela (18 years), Walter Sisulu, Govan Mbeki, Ahmed Kathrada, Robert Sobukwe, Jacob Zuma (10 years), and thousands of others. The tour ends at Cell Block B, Cell 5 – the 2.4 × 2.1-metre concrete room in which Mandela spent those eighteen years.

Tours are run by the Robben Island Museum and depart from the Nelson Mandela Gateway at the V&A Waterfront at 09:00, 11:00, 13:00, and 15:00. The experience takes 3.5 to 4 hours including the 30-minute ferry crossing each way. The bus tour of the island is narrated by a trained guide; the prison walking tour itself, crucially, is led by a former political prisoner. This is the single most distinctive feature of the Robben Island experience and the reason it remains one of the most emotionally powerful heritage visits anywhere in the world.