A game called Suffering...I don't play that many video games, but it was pretty scary at some points..especially when the monsters would jump from the ceiling and stab the police officers in the head or other limbs and leave a bloody mess. Rated M, no doubt.
I would have to say The original silent hill was the scariest due to the fact that I was Younger when i played it so it kinda got to me.. or that Fatal Frame game.. the one where you are this little chinese girl and the only way you could kill the ghost was to take pictures of it. it was a very freaky game