TLS/SSL Certificate Authority (CA) Trust store verification

The CA trust store is how a TLS client establish trust with the servers offered certificates. Curl uses openssl and Python uses its own store (like Java). The store on my system is located here:

~/AppData/Roaming/Python/Python37/site-packages/certifi/cacert.pem

You can check which store your Python is using via the command line:

Python 3.8.5 (default, Jul 28 2020, 12:59:40)
>>> import certifi
>>> certifi.where()
'/etc/ssl/certs/ca-certificates.crt'

I was having trouble with a proxy farm where one of the proxy (which is doing MITM) was sending a random non trusted certificate. This script was written to keep trying until it found the bad proxy. The problem is openssl is used in the backend by python and its interface only exposes the bottom certificate of the chain. So this is fairly useless if the server is offering a few certificates. Script is kept here for that one day in the future when they fix the bug (https://bugs.python.org/issue18233). The latest PR to fix it is https://github.com/python/cpython/pull/17938.

It seems like a very contentious issue, in my case I want the raw peer cert list without the validation / security / smarts but the Python developers seem to think that they should be doing the validation up the chain before returning it to the user. In this case they need to take some of Kimi Raikkonen’s advice “leave me alone, I know what I’m doing”

import ssl
from urllib.parse import urlparse
import click
import socket
import base64

def get_certificate(url: str) -> str:
   """Download certificate from remote server.
   Args:
      url (str): url to get certificate from
   Returns:
      str: certificate string in PEM format
   """
   parsed_url = urlparse(url)
   hostname = parsed_url.hostname
   port = int(parsed_url.port or 443)
   conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

   message = b'usernamehere:passwordhere'
   prox_auth = base64.b64encode(message)



   headers = """CONNECT {}:{} HTTP/1.1
Proxy-Authorization: Basic {}
Host: {}\r\n\r\n""".format(hostname, port, prox_auth.decode("ascii"), hostname)


   print(headers)
   proxy_host = "proxy" #proxy server IP
   proxy_port = 8080              #proxy server port
   conn.connect((proxy_host, proxy_port))
   conn.send(headers.encode('utf-8'))
   other = conn.recv(30000000)
   # print(other)


   context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
   # context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
   sock = context.wrap_socket(conn, server_hostname=hostname)
   # sock.connect((hostname, port))
   return ssl.DER_cert_to_PEM_cert(sock.getpeercert(True))

@click.command()
@click.argument("url", type=str)
def main(url: str):
   """Retrieve and print out the ssl certificate.
   Args:
      URL (str): url to be picked up
   """
   click.echo(get_certificate(url), nl=False)

if __name__ == "__main__":
   main()

Who decides what certificate authorities are publicly trusted?

  • Here is how Mozilla decides which root certificates to bundle with Firefox:

  • Curl has a script that updates the Mozilla CA cert file daily:

Trust store on different Linux systems

Comments

comments powered by Disqus