Authentication in OSPF isn’t all that mysterious. But there are a few points that can trip people up. Here are 3 different things to be aware of. All of these tips are on the IP Expert BLS videos.
Authenticating with an unspecified password
Regardless of if you use plain text authentication or an MD5 hash, you can actually enable authentication, not specify a password, and have the neighbors stay up. For instance, setup OSPF and get your neighbors up. Go to one of your interfaces and enter the “ip ospf authentication” command. Eventually, the neighbor on the other side of the link will go away. Configure the same command on the other side of the link, and the neighbor will come back up. Even though you didn’t specify a password.
Now this can get you into trouble if the lab says to use a specific password. You might forget, but everything will continue to work. So you think you got the points where you didn’t.
Another pitfall that falls in this category is if the lab asks you to use md5 authentication with a password of Cisco. So you configure “ip ospf authentication message-digest” and “ip ospf authentication-key cisco”. Everything comes up, and you think you got the points. Well, you didn’t because the ip ospf authentication-key specifies a clear-text password. To configure the md5 password, you need to use “ip ospf message-digest-key” command. So on the lab, be careful that you are entering the correct commands and properly verifying using the “show ip ospf interface” command.
MD5 key number mismatch
Another tip that you might not know is that the MD5 key numbers need to match in addition to the password. So if the interface on one side of the link uses key 1, and the interface on the other side used key 2, the routers will not become neighbors. If you do a debug, you’ll see the following message.
*Mar 1 00:19:44.943: OSPF: Rcv pkt from 10.10.20.1, Serial0/0 : Mismatch Authentication Key – No message digest key 2 on interface
Area 0 authentication and virtual links
One last tip comes when area 0 is setup for authentication and you use virtual links. If you run into this scenario, you need to configure authentication on the virtual link. If you don’t, you’ll configure the virtual link, but nothing will come up. Debugs will show the message below on the far end router.
*Mar 1 00:34:36.771: OSPF: Rcv pkt from 10.10.20.1, OSPF_VL0 : Mismatch Authentication type. Input packet specified type 1, we use type 0
The configuration needs to be entered on the virtual link using “area x virtual-link x.x.x.x authentication” commands.