The novel provides a detailed, episodic record of life in the two branches of the wealthy, aristocratic Jia (賈) clan—the Rongguo House (榮國府) and the Ningguo House (寧國府)—who reside in two large, adjacent family compounds in the capital. Their ancestors were made Dukes and given imperial titles, and as the novel begins the two houses are among the most illustrious families in the city. One of the clan’s offspring is made a Royal Consort, and a lush landscaped garden is built to receive her visit. The novel describes the Jias’ wealth and influence in great naturalistic detail, and charts the Jias’ fall from the height of their prestige, following some thirty main characters and over four hundred minor ones. Eventually the Jia clan falls into disfavor with the Emperor, and their mansions are raided and confiscated.

In the novel's frame story, a sentient Stone, abandoned by the goddess Nüwa when she mended the heavens aeons ago, begs a Taoist priest and a Buddhist monk to bring it with them to see the world. The Stone, along with a companion (in Cheng-Gao versions they are merged into the same character), is then given a chance to learn from the human existence, and enters the mortal realm.

The main character of the novel is the carefree adolescent male heir of the family, Jia Baoyu. He was born with a magical piece of "jade" in his mouth. In this life he has a special bond with his sickly cousin Lin Daiyu, who shares his love of music and poetry. Baoyu, however, is predestined to marry another cousin, Xue Baochai, whose grace and intelligence exemplify an ideal woman, but with whom he lacks an emotional connection. The romantic rivalry and friendship among the three characters against the backdrop of the family's declining fortunes form the main story in the novel.