First, you will need my command line utility. It's a trivial thing that only returns a status code based on whether there was some standard input piped in. AnyStdIn.zip This will only work on 64-bit OS-es (Windows). Make sure it's on your Path system variable.

Second, you need to modify your .hg/hgrc file inside the repository that you want this hook to be relevant to. I prefer to have this for all repos, so I put it inside mercurial.ini file. What this does is creates a "hook" that will fire before the commit event, it will list all "untracked files", and if there are any they will be piped into AnyStdIn.exe, which will return status code 1 and abort the commit.

[hooks]
precommit.No_Untracked_Files = hg st -u | AnyStdIn

And... that's it. If there are untracked files, the message will be something like:

abort: precommit.No_Untracked_Files hook exited with status 1

If for any reason you'd want to avoid this check you'd have to temporarily comment this hook's line.