Call of Cthulhu
The originator of the sanity point system as a parallel mental damage track to the traditional hit point. It deals in cosmic horror based primarily on the writing of Lovecraft and his disciples and as such posits that the human mind is fundamentally unequipped to deal with the reality of its insignificance in the grand scheme of the universe.
Arkham Horror/Eldar Sign/Mansions of Madness
Board games based on Lovecraft and using a simplified version of Call of Cthulhu's parallel health and sanity scores, although generally less deadly.
Urban/religious horror. Insanity is modelled as degradation and often depravity, but also as insight into the true nature of reality. Insanity is modelled as progressive, situated upon a roadmap marking the character's distance from 'normality', and is complemented by an alternative roadmap which represents a sort of hyper-sanity.
Clumsy 'the world's gone mad' dystopian sci-fi. Everyone has a discrete, diagnosed condition and there's a marble-based resolution mechanic. There's a good idea in there somewhere though.
AD&D: Domains of Dread
Ravenloft setting-specific rules. Introduced a horror system via graded saving-throws dependent on severity of stimulus, the end result of which could be a randomly rolled psychiatric condition. As is often the case with modular systems, despite good intentions and a clear attempt to research the matter at hand the tendency towards AD&Dness of the resulting rules leaves them clunky and in this case rather offensive.
A system of feedback loops based on failing or passing checks testing how well they can cope with stressful and horrific situations can cause certain propensities of a character to become more pronounced and even to develop into a psychiatric condition. Remarkably subtle, though still ultimately wedded to a very diagnostic model of mental illness. John Tynes had previously published the meta-game/polemical essay Power Kill which is rather more simplistic and stigmatising.
The World of Darkness
Various editions and games approach it differently, and in some iterations it is more hard-coded in the rules than in others. I have only ever played the 1st-3rd Edition rules, although I've read that the reboot contained major changes. In general there is a system tracking how far from human the monstrous player-characters fall, and while this is mostly a dramatic system played for pathos it is also used for mental resilience. Introduced irrevocably 'mad' player-characters as, effectively, character classes as well as the option to take psychiatric conditions as flaws for bonus points.
Warhammer Fantasy Role Play
Insanity is an optional rule, it is also a positive measure - characters gain insanity points from traumatic events which may then turn into psychiatric conditions in contrast to the majority of games where mental resilience is eroded. this works well with the theme of a hostile world in which anyone can fall.
The Victory of Joan of Arc
Historical simulation of the siege of Orleans, which I've written about here. Mechanics make an implicit link between madness, divine inspiration and irrationality.
Eternal Darkness: Sanity's Requiem
As a sanity gauge falls, glitchy and gimmicky 'sanity effects' disrupt play. Really quite crass.
A gauge measures the sanity/resilience of the characters, dropping as the testing events of the game take their toll and raising as the characters achieve successes but also do normal things. Hitting zero provides a game over, either by suicide or otherwise giving up. The game is really about the contrast between the supernatural horror of the story and normal life, and this mechanic reinforces the healing effects of normality. I hear that the sequel Heavy Rain might also do something similar, but I have no way of playing it at the moment.
Models depression in a twine game by restricting options, which is a nice mechanic. Very prescriptive on how to beat it though. I wrote about it here.
Mental Illness as a Theme without Direct Modelling of Sanity
Batman: Arkham Asylum/Arkham City
Foucault the video game, with added punching. Batman is a failed panopticon installed in the head of criminals, turning them mad, and can only exist fully himself whilst incarcerated. The lines between who is and who isn't mad are hopelessly blurred. I love these games.
Warhammer Fantasy battle/Warhammer 40,000
Chaos is a huge part of the Warhammer fluff, and so while you can play the games without ever worrying about it the very metaphysics that underpin these worlds - and drive them into the paroxysms of war that support the huge gaming space they encompass - is one of madness. It is actually the reverse of real-world physics, as rather than entropy the universe is feeding back on itself to generate a chaotic state.
Is he mad or isn't he? Has a deep distrust of psychiatry and institutionalisation - riffing heavily off One Flew Over the Cuckoo's Nest.
I've only ever played a few hours of the first one, but I am reliably informed that there is something self-created in the hellish nature of Silent Hill.
Satirical, sci-fi splatterpunk primarily played for laughs. The game's main theme is the corruption of total power and its effect on social norms. The characters act in an aggressively sane manner given their situation, they just seem mad to an outsider, raising questions about the socially constructed nature of definitions of madness.
The actions of insane characters and of a corrupt psychiatrist are key to the plot. I wrote about it extensively here.
I haven't played either game, but reviews etc. claim that everyone (?) in the world is supposed to be non-specifically insane. Also contains a character explicitly diagnosed with Asperger's.