China says some 300,000 civilians were massacred in the city after its occupation by Japanese troops in 1937, although Japan disputes this.
The ceremony on the 77th anniversary of the massacre is part of three new public holidays intended to mark Japan's conflict with China.
Relations between the two countries have been strained in recent years.
They have clashed over island territory in the East China Sea as well as over Japan's insistence on honouring its war dead, including convicted war criminals, at the Yasukuni shrine.
Deniers criticised
At the ceremony in Nanjing, about 10,000 participants stood in silence for one minute to honour those killed.
They included survivors of the massacre, as well as soldiers and students.
In a speech at the event, Mr Xi criticised Japanese nationalists for seeking to deny the atrocity took place.
"Those who uphold justice and love peace must be highly vigilant and firmly oppose those wrong words and deeds," he said.
The event was designed to "arouse every kind person's longing for and adherence to peace, and not to perpetuate hatred", Mr Xi added.
Millions of Chinese people were killed when Japan occupied China in the 1930s and 1940s.
