The film opens showing an unidentified woman bathing in a hot spring as an unidentified man describes a woman's skin.Hervé (Michael Pitt), in 19th century 1860s France, is a military officer, because that is what his father wants. He is happy when he gets the opportunity to change his profession from enlisted soldier to that of trader of silkworm eggs. Because in Europe silkworms are affected by a disease (pébrine), he is sent to Africa to get healthy eggs. (The film shows only a brief desert scene.)Since the African silkworms are affected too, he is sent to Japan, even though at that time (the Bakumatsu period) almost all of Japan was off-limits to foreigners. The journey is by carriage and train, a caravan of horses through Europe, then Asia, the boat trip from the Asian continent to Japan, and guided, blindfolded, on horseback to the Japanese village where he can buy the eggs from a local baron, Hara Jubei (Koji Yakusho).During his stay in the village, Hervé becomes obsessed with Jubei's unnamed concubine (Sei Ashina). On his second journey there, she gives him a note in Japanese, and he has sex with another girl handed to him by her. Having traded more eggs than on his first trip, Hervé delays his departure by two days in the hope of seeing the girl again.Back home in France, Hervé's love obsession to the nameless Japanese girl remains secret, and though he seems happy with his wife, Hélène (Keira Knightley), they remain childless. Hervé seeks out a French-Japanese brothel owner in Lyon, Madame Blanche (Miki Nakatani), known for giving the small blue flowers that she wears to her clients, who translates the note for him as: "Come back or I shall die". Madame Blanche advises Hervé to "forget about her, she won't die".During Hervé's third trip to Asia, the silk entrepreneur Baldabiou (Alfred Molina) wants to send him to China, since Japan is no longer safe, and good eggs are now available in China. Regardless of the politics in Japan, Hervé insists on going there anyway. When he arrives in Japan, however, war has broken out and the village is abandoned; Jubei and the girl are not there. Jubei's young servant boy (Kanata Hongo), whom Hervé has met before, shows Hervé where Jubei and his household have gone. When Jubei sees Hervé he becomes hostile and tells Hervé to go home to France. Before he leaves, Hervé discovers that Jubei had the boy executed (by hanging) for showing Hervé where to find them. Without seeing the girl, Hervé leaves Japan, managing to buy some silk eggs in Sakata, and returns home. Unfortunately his side trip has delayed his departure until too late in the season, and before reaching France Hervé discovers that the eggs have hatched and all the silk caterpillars have died. As a result, three of the silk mills are forced to close down. To employ the village, Hervé decides to build Hélène's garden instead.Months later, Hervé receives a long letter from the mysterious concubine, written in Japanese. He again takes the letter to Madame Blanche for translation, who agrees, providing Hervé never comes to see her again. The letter is a deeply moving declaration of love.Hélène becomes ill and eventually dies. After her death, Hervé finds a tribute of small, blue flowers on her grave. He seeks out Madame Blanche once more, believing her to have written the letter, but Madame Blanche reveals to him that Hélène had written the letter. Hélène had asked Madame Blanche to translate the letter into Japanese, knowing that her husband was in love with a Japanese woman, and wanting him to be happy. Madame Blanche tells Hervé that, more than anything, his wife wanted to be that woman. Too late, Hervé finally realizes that it was Hélène who was his true love after all.The film ends with Hervé, fated to remain childless and alone, recounting his story to Ludovic (Mark Rendall) - a young protege since childhood and now Hélène's gardener - on a bench in Hélène's garden. Henceforth, Ludovic can now appreciate the real story behind Hélène's garden.