A problem seen fairly often on directly connected serial lines
(mgetty -r), and seldomly on modem lines, is that mgetty
works flawlessly, but /bin/login just hangs instead of prompting for
the user password.
The reason for this is that many login programs reopen
/dev/tty (the controlling tty of a process) to make sure they have
full control over the password entered by the user (for example, to prevent
snooping). This will block on some systems if the DCD (carrier detect) line
coming from the modem or the other machine is low. Notably those systems
are those that have callin/callout device pairs for one serial device, e.g.
Linux, SunOS, SCO/FAS, etc.
The fix is easy: make sure that the DCD line is high.
If you use a modem, the command to do this is usually AT&C1 (but
check with your modem manual).
If you’re using a direct null-modem connection to another host, the recommended solution is to wire DCD on your side to the DTR line on the other side and vice versa. That way, when the remote machine “hangs up” (calling program exits and DTR drops), your host will get notified as well. This is what a properly wired (!) “null-modem cable” does anyway.
If you don’t have free lines in your serial cable (classic three-wire
approach), wire DCD to the DTR line on your own host, and make sure that
mgetty won’t toggle DTR upon startup (causing a hangup signal!), e.g. by
setting toggle-dtr no in mgetty.config.