Sunday, 30 July 2017

CURL issue with NSS 3.28.4-3.0.1 - curl: (35) SSL connect error - ciphers rsa_rc4_128 only working

Problem Statement

Recently when a Linux monitoring script using CURL to monitor a https URL broke when migrated to a VM from Oracle Linux v6.7 to v6.9. This post i will try to explain the root cause of the issue and solution to fix the issue.

Solution

  • Oracle Linux v6.7 the following command was working fine: VM-1$curl <your-https-server-url>
  • But in Oracle Linux v6.9 the same command was failing: VM-2$curl <your-https-server-url>
    • The error it was throwing: "curl: (35) SSL connect error"
  • The supported cipher for my server was found to be:
  Supported Server Cipher(s):
Preferred TLSv1.0 128 bits DHE-RSA-AES128-SHA DHE 768 bits
Accepted TLSv1.0 128 bits AES128-SHA
Accepted TLSv1.0 128 bits RC4-SHA
  • By googling about the error i found in some post to use following -cipher option with curl and specify the cipher explicitly for the things to work:
VM-2$curl --ciphers rsa_rc4_128_sha <your-https-server-url>
  • The following are debugging steps i followed to debug the issue further:
  • Check the meaning of error "curl: (35) SSL connect error"
VM-2$curl -v <your-https-server-url>
* About to connect() to sas1.sal.avaya.com port 443 (#0)
*   Trying 135.11.105.105... connected
* Connected to sas1.sal.avaya.com (135.11.105.105) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12173
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error
  • Now go to the URL https://mozilla.github.io/python-nss-docs/nss.error-module.html and check the meaning of the error "NSS error -12173"
  • The error is "SSL_ERROR_WEAK_SERVER_EPHEMERAL_DH_KEY: SSL received a weak ephemeral Diffie-Hellman key in Server Key Exchange handshake message."
  • Now check the CURL version in the two VMs:
VM1$ curl --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

VM2$ curl --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
  • If you observe the difference in the NSS library being used by CURL. So next step is to look what is different in NSS.
VM1$rpm -qa | grep nss

VM2$rpm -qa | grep nss
  • The difference i noted is on the following rpm: 
  • "nss-3.28.4-3.0.1.el6_9.x86_64" versus "nss-3.28.4-1.0.0.1.el7_3.x86_64"
  • So the time is to now check what has got changed between the two version
VM1$rpm -ql nss
/etc/pki/nssdb
/etc/pki/nssdb/cert8.db
/etc/pki/nssdb/key3.db
/etc/pki/nssdb/secmod.db
/usr/lib64/libnss3.so
/usr/lib64/libnssckbi.so
/usr/lib64/libnsspem.so
/usr/lib64/libsmime3.so
/usr/lib64/libssl3.so
/usr/lib64/nss/libnssckbi.so


VM2$rpm -ql nss
/etc/pki/nss-legacy
/etc/pki/nss-legacy/nss-rhel6.config
/etc/pki/nssdb
/etc/pki/nssdb/cert8.db
/etc/pki/nssdb/key3.db
/etc/pki/nssdb/secmod.db
/usr/lib64/libnss3.so
/usr/lib64/libnssckbi.so
/usr/lib64/libnsspem.so
/usr/lib64/libsmime3.so
/usr/lib64/libssl3.so
/usr/lib64/nss/libnssckbi.so


  • One of the observation is a new config file "nss-rhel6.config" which got introduced and if you see its content:
VM2$cat /etc/pki/nss-legacy/nss-rhel6.config
# To re-enable legacy algorithms, edit this file
# Note that the last empty line in this file must be preserved
library=
name=Policy
NSS=flags=policyOnly,moduleDB
config="disallow=md5 allow=DH-MIN=1023:DSA-MIN=1023:RSA-MIN=1023"

  • Now you can correlate the error SSL_ERROR_WEAK_SERVER_EPHEMERAL_DH_KEY we are getting and the last line in above config file mentioning DH key size 1023 as minimum supported.
  • This gives an indication that the DH key size restriction has been introduced in newer version of NSS,
  • So I did give a try by reducing the key size to 767 (which i got from checking the DH size being handshaked in VM1.
VM2$vi /etc/pki/nss-legacy/nss-rhel6.config
# To re-enable legacy algorithms, edit this file
# Note that the last empty line in this file must be preserved
library=
name=Policy
NSS=flags=policyOnly,moduleDB
config="disallow=md5 allow=DH-MIN=767:DSA-MIN=767:RSA-MIN=767"

  • This change made the curl in VM1 to work without specifying the -cihper option explicitly.

Conclusion

The lastest version of NSS library nss-3.28.4-3.0.1.el6_9.x86_64 has tighten the ssl handshake by putting restriction to allow DH min key size of 1024. Which is the best security practice but if you want to make the things working without changing the Server TLS version and cipher supports then this is the way to handle it.

Thanks!