I am trying to generate PKCS12 public and private keys with OpenSSL on Windows 7 64-bit that can be used by both the Microsoft CryptoAPI in C as well as Java programs.
Here are the steps I have followed:
Downloaded installed Microsoft Visual C++ 2008 Redistributable Package (x64) de: http://www.microsoft.com/en-us/download/details.aspx?id=15336
Downloaded installed "Win64 OpenSSL v1.0.2a Light" de: http://slproweb.com/products/Win32OpenSSL.html
To encrypt/decrypt files of arbitrary size using asymmetric (public) key cryptography you need to use S/MIME encoding:
1) generate the key pair. This makes a 2048 bit public encryption key/certificate rsakpubcert.key and a matching private decryption key rsakpriv.key. The -days 10000 means keep it valid for a long time (27 years or so). You will be asked (twice) for a PEM passphrase to encrypt the private key. If you do not wish to encrypt it, pass the -nodes option. The public key can be distributed to anyone who wants to send you data.
md C:\OpenSSL-Win64\bin    
cd C:\OpenSSL-Win64\bin    
set OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg    
openssl req -x509 -days 10000 -newkey rsa:2048 -keyout c:\opensslkeys\rsakpriv.key -out c:\opensslkeys\rsakpubcert.key
// pass phrase used: mypassword
2) Create request for self-signed certificate
openssl req -new -key c:\opensslkeys\rsakpriv.key -out c:\opensslkeys\server.csr
3) Remove password from the private key
openssl rsa -in c:\opensslkeys\rsakpriv.key -out c:\opensslkeys\rsakprivnopassword.key
4) Self-sign the certificate request (-days is expiration in days)
Important: start the command prompt with "Run As Administrator". Otherwise you get the same: unable to write 'random state' error.
cd C:\OpenSSL-Win64\bin    
set OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg    
openssl x509 -req -days 365 -in c:\opensslkeys\server.csr -signkey c:\opensslkeys\rsakprivnopassword.key -out c:\opensslkeys\server.crt
5) Convert output to PKCS#12 format which we can use in code (-keysig parameter allows us to use key-pair for signing)
openssl pkcs12 -export -in c:\opensslkeys\server.crt -inkey c:\opensslkeys\rsakprivnopassword.key -out c:\opensslkeys\mypublicencryptionkey.p12    
// export password used: mypassword
At this point I can encrypt and decrypt files with openssl with these commands:
To encrypt:
openssl smime -encrypt -binary -aes-256-cbc -in c:\opensslkeys\todo.txt -out c:\opensslkeys\done.txt -outform DER c:\opensslkeys\server.crt
To decrypt:
openssl smime -decrypt -binary -in c:\opensslkeys\done.txt -inform DER -out c:\opensslkeys\redone.txt -inkey c:\opensslkeys\rsakprivnopassword.key
However when I try to use the keys in Java the program chokes, saying the key format is wrong.
Any help greatly appreciated! I promise to post the com,plete working answer for others.
Complete Java code:
/*
PrivatePublicKey.java   
*/
import java.io.FileInputStream;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.PublicKey;
import java.security.cert.Certificate;
import javax.crypto.Cipher;
/* for IBM JDK need to replace: */
//import java.util.Base64;
/* with: */
import org.apache.commons.codec.binary.Base64;
public class PrivatePublicKey 
{
    public static void main(String[] args) throws Exception 
    {
        try 
        {
            PublicKeyReader myPublic = new PublicKeyReader();
            PublicKey publicKey = myPublic.get("./rsakpubcert.key");
            PrivateKeyReader myPrivate = new PrivateKeyReader();
            PrivateKey privateKey = myPrivate.get("./rsakprivnopassword.key");
            // Let's encrypt with private and decrypt with public
            // Encrypt with private key
            String firstString = "Ishana";
            Cipher privateEncryptCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
            privateEncryptCipher.init(Cipher.ENCRYPT_MODE, privateKey);
            byte[] encryptedFirstString = privateEncryptCipher.doFinal(firstString.getBytes());
            String encodedEncryptedFirstString = Base64.encodeBase64String(encryptedFirstString);
            System.out.println("Encoded encrypted String for Ishana: " + encodedEncryptedFirstString);
            // Decrypt with public key
            // First decode the string
            byte[] decodedEncryptedFirstString = Base64.decodeBase64(encodedEncryptedFirstString);
            Cipher publicDecryptCipher = Cipher
                .getInstance("RSA/ECB/PKCS1Padding");
            publicDecryptCipher.init(Cipher.DECRYPT_MODE, publicKey);
            byte[] decryptedFirstStringByte =     publicDecryptCipher.doFinal(decodedEncryptedFirstString);
            System.out.println("Decrypted String for Ishana: " + new String(decryptedFirstStringByte));
        }
        catch (Exception e) 
        {
            e.printStackTrace();
        }
    }
}
Edit, added helper code:
import java.io.*;
import java.security.*;
import java.security.spec.*;
public class PublicKeyReader {
  public static PublicKey get(String filename)
throws Exception {
    File f = new File(filename);
    FileInputStream fis = new FileInputStream(f);
    DataInputStream dis = new DataInputStream(fis);
    byte[] keyBytes = new byte[(int)f.length()];
    dis.readFully(keyBytes);
    dis.close();
    X509EncodedKeySpec spec =
      new X509EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePublic(spec);
  }
}
and
import java.io.*;
import java.security.*;
import java.security.spec.*;
public class PrivateKeyReader {
  public static PrivateKey get(String filename)
  throws Exception {
    File f = new File(filename);
    FileInputStream fis = new FileInputStream(f);
    DataInputStream dis = new DataInputStream(fis);
    byte[] keyBytes = new byte[(int)f.length()];
    dis.readFully(keyBytes);
    dis.close();
    PKCS8EncodedKeySpec spec =
      new PKCS8EncodedKeySpec(keyBytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    return kf.generatePrivate(spec);
  }
}
 
     
    