Mutual authentication is a basic concept. The client and service must prove their respective identities to each other before performing application functions. The central principal of mutual authentication is that neither party must "trust" the other before identity has been proven; that is, the service must be able to determine who the client is without querying the client and the client must be able to determine who the service is without querying the service.
The value of a service that can authenticate a client is well-known. For example, a file service impersonates a client's identity to determine which files the client is allowed to access.
The value of a client being able to authenticate a service is less understood. Authenticating a service enables the client to trust the data that it gets from the service and to feel secure in sending sensitive data to the service. The ability of a client to authenticate a service is particularly important in client/service applications that support delegation of the client's security context; that is, the client authorizes the service to act as its delegate in accessing additional services or network resources.
A service authenticates a client as follows: The client establishes a local security context either by executing in a previously established context, for example, in the session of a logged-in user, or by explicitly presenting credentials to the underlying security provider. The service will not accept connections from any unauthenticated client.
The Kerberos mechanism by which a client authenticates a service works as follows: When a service is installed, a service installer, running with administrator privileges, registers one or more unique SPNs for each service instance. The names are registered in Active Directory on the user or computer account object that the service instance will use to log on. When a client requests a connection to a service, it composes an SPN for a service instance, using known data or data provided by the user. The client then uses the SSPI negotiate package to present the SPN to the Key Distribution Center (KDC) for the client domain account. The KDC searches the forest for a user or computer account on which that SPN is registered. If the SPN is registered on more than one account, the authentication fails. Otherwise, the KDC encrypts a message using the password of the account on which the SPN was registered. The KDC passes this encrypted message to the client, which in turn passes it to the service instance. The service uses the SSPI negotiate package to decrypt the message, which it passes back to the client and on to the client's KDC. The KDC authenticates the service if the decrypted message matches its original message.