Windows 2012 서버에서 Transport Layer Security (TLS) Handshake Failing, Schannel error 36888 에러 발생 시 조치 사항
최근 구성한 Windows 2012 OS 환경에 ADDC를 구성하고 외부 서비스에서 LDAPs 포트 636 and 3269 를 통해 인증 서비스를 테스트 하는 과정에서 많이는 아니지만, 자주 아래 에러를 발생하면서 인증이 안되는 문제가 확인되어 문제 확인 결과 아래와 같이 같이 Register key 변경을 통해 문제 해결이 가능함을 확인 했습니다.
아래 원문 기준으로 Windows 2008R2, 2012R2 버전에서 무조건 발생하는 문제로 보여 집니다. AD Join 클라이언트에서 발생하는 문제는 아니고, LDAP 포트를 이용한 인증 서비스 연계 시 발생하는 문제로 봐야하는게 맞을 듯 합니다.
Scenario
The scenario that seems to be the most common is the customer has a 3rd party device or application that is connecting to a Microsoft Server (2008R2, 2012, and 2012R2 are the most common) and they are either failing the TLS handshake completely or sporadically.
Cause
The first thing that you should do is gather a network trace that lines up with the errors you are seeing. Use the network capture tool that you feel most comfortable with. Drilling down into the network trace to the Server Hello will tell us the Cipher Suite that is being negotiated. In the example below, and in most of the cases I have seen, the problematic Cipher Suite will be either TLS_DHE_RSA_WITH_AES_128_CBC_SHA or TLS_DHE_RSA_WITH_AES_256_CBC_SHA. There are various interoperability and deeply technical reasons for this that I will not get into in this article. Keep in mind that I have seen cases where a Cipher Suite other than the two above was the culprit so you may still want to follow my recommendations for remediation even if that is the case.
Resolution
The Cipher Suite order is determined by one of two registry keys
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002
or
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002
The second one is set by either a Local Policy or a Group Policy and takes precedence over anything defined in the first one. If you do not have a policy in place it will be blank.
Opening the Functions key will display a Multi String valued list that looks like below:
Disclaimer: Please backup this key before making any changes to it so that you can revert to back if necessary.
If the Cipher Suite you are seeing in your network trace is one of the two I outlined above, I would recommend simply cutting both of them out and then saving the registry key so that they are no longer on the list. A reboot MUST be done after this change for it to take effect. Doing this is a relatively quick and easy way to determine if the Cipher Suites are the issue.
Did that resolve your issue? If so, great. As a supported long term solution you really want to set the Cipher Suites via either a Local policy or a Group Policy so that future OS hotfixes will not add the Cipher Suites back in and return it to a broken state. Please refer to the following article for instructions on setting this in Group Policy: https://msdn.microsoft.com/en-us/library/windows/desktop/bb870930(v=vs.85).aspx#adding__removing__and_prioritizing_cipher_suites
Additionally it is recommended that you patch the OS fully when possible.
If this did not resolve your issue, your next step would be to move the below highlighted Cipher Suites to the top of the order (they are the ones that start with TLS_RSA_WITH_AES. Cut them from wherever they are in the order, paste them at the top, save the key, then reboot the server. (Again, a reboot is a MUST)!
Did that resolve the issue? If so, great. Please follow the advice stated previously for setting it via Group Policy as a long term supported solution.
If this did not resolve your issue it may be time to open a support call with us to assist you in figuring it out. Microsoft Support will ask you to gather data. I wrote the following blog to assist with that and it helps in cases where the issue is sporadic and not easy to reproduce.
Either way it should gather what we will initially need to analyze and diagnose and you will be one step ahead of the game.
Cheers and happy hunting!
