I know I can view decrypted HTTPS data directly in Charles. But I want to view lower level SSL messages in Wireshark. So I setup Charles as the SSL proxy, using my own certificate, and I setup SSL dissector preference in Wireshark with local IP address and my private key.
However, the messages in Wireshark are still encrypted.
What did I miss?
I encountered the exact same problem with Charles Proxy in combination with Wireshark.
I think the issue is that Charles sends two (or more) certificates to the client (check the Certificate message sent from proxy to the client). Wireshark will then use the first certificate in that list, which probably will not match the private key you generated.
(This is exactly what user dave_thompson_85 is wondering in the comments.)
I have checked this by extracting the certificate from Wireshark. Note that Wireshark extracts the certificate in .der format. Then I have converted the .der-file to a .pem certificate:
openssl x509 -inform DER -outform PEM -text -in wireshark_charles.der -out wireshark_charles.pem
I also have converted the .pem to a .crt, but this is not necessary.
Certificate sent by Charles to client
$ openssl x509 -noout -modulus -in wireshark_charles.crt | openssl md5
7a37a32781daf79402623c19ac9c8d7f
Custom certificate set up in Charles
$ openssl x509 -noout -modulus -in charles_custom.crt | openssl md5
62ea5ed061fca62efaaecbbb0226b08e
The corresponding private key
$ openssl rsa -noout -modulus -in charles_custom.pem | openssl md5
62ea5ed061fca62efaaecbbb0226b08e
The modulus of the certificate sent by Charles does not match the modulus of the custom generated private key.
And Wireshark also logs this issue during the SSL Dissection:
ssl_decrypt_pre_master_secret wrong pre_master_secret length (128, expected 48)
ssl_generate_pre_master_secret: can't decrypt pre master secret
Charles generates a new per-host certificate using the custom certificate as root certificate. Unfortunately, I have not found a way to extract this per-host private key generated by Charles. I suggest to use Burp Proxy. In Burp you can select which type of certificate you want to use.
