Java Decryption of openssl aes-256-ctr encrypted file

Question

I try to decrypt in java a file, which was encrypted via openssl:

openssl enc -aes-256-ctr -in raw.zip -out encrypted.zip.enc -pass stdin

My implementation looks currently terrible, because it is just a scratch.

 public static void main(String[] args)
    throws IOException, NoSuchAlgorithmException, InvalidKeySpecException, NoSuchPaddingException, InvalidKeyException, InvalidAlgorithmParameterException {

    FileInputStream fis = new FileInputStream(new File("/tmp/encrypted.zip.enc"));
    /* Derive the key, given password and salt. */
    SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
    byte[] salt = new byte[8];
    fis.read(salt, 0, 8);// Salted__
    fis.read(salt, 0, 8);// real Salt

    KeySpec spec = new PBEKeySpec("myPassphrase".toCharArray(), salt, 65536, 256);
    SecretKey secret = new SecretKeySpec(factory.generateSecret(spec).getEncoded(), "AES"); 

    // build the initialization vector.  This example is all zeros, but it
    // could be any value or generated using a random number generator.
    byte[] iv = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };
    IvParameterSpec ivspec = new IvParameterSpec(iv);
    Cipher cipher = Cipher.getInstance("AES/CTR/PKCS5PADDING");
    cipher.init(Cipher.DECRYPT_MODE, secret, ivspec);
    CipherInputStream inputStream = new CipherInputStream(fis, cipher);
    FileOutputStream fos = new FileOutputStream(new File("/tmp/decrypted.zip"));

    byte[] buffer = new byte[1024];
    int len;
    while ((len = inputStream.read(buffer)) != -1) {
        fos.write(buffer, 0, len);
    }
}

The file is not the same like before. the hashes differs. I guess, that there is a problem with the secret key. Is it right? Should I use another instances?

Where did you copy your implementation from? Perhaps this [answer](https://stackoverflow.com/a/992413/3301492) might be helpful. — Boris, Sep 19 '19 at 14:27
@Boris: that's for password-based in general but not compatible with openssl specifically (unless openssl 1.1.1 with -pbkdf2 as Zergatul says). Instead see https://stackoverflow.com/questions/14695766/ https://stackoverflow.com/questions/11783062/ https://stackoverflow.com/questions/31947256/ https://stackoverflow.com/questions/32508961/ https://stackoverflow.com/questions/29151211/ — dave_thompson_085, Sep 19 '19 at 23:42

score 0 · Answer 1 · answered Sep 19 '19 at 16:49

If you use openssl enc without -pbkdf2 it uses some internal key derivation function. My example:

openssl command:

openssl enc -aes-256-ctr -in 1.txt -out 1.enc -pbkdf2 -iter 1000 -pass pass:qwerty -p

Parameter -p outputs generated keys. In my case it was:

salt=063C06BA16675384
key=45743B02A171425197014D80A08D1024CD97587272BAEBE1F1F0FC3AC0164AB2
iv =9DF736227817CBEC9DF5397F1C05F31A

Java code:

public static void main(String[] args) throws Exception {

    FileInputStream fis = new FileInputStream(new File("1.enc"));

    fis.skip(8);
    byte[] salt = new byte[8];
    fis.read(salt);

    System.out.println("salt=" + byteArrayToHex(salt));

    // you used PBKDF2 with SHA1, but openssl uses SHA256
    SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
    // you should generate 48 bytes, key and IV
    KeySpec spec = new PBEKeySpec("qwerty".toCharArray(), salt, 1000, 48 * 8);
    byte[] bytes = factory.generateSecret(spec).getEncoded();

    byte[] key = Arrays.copyOfRange(bytes, 0, 32);
    byte[] nonce = Arrays.copyOfRange(bytes, 32, 48);

    System.out.println("key=" + byteArrayToHex(key));
    System.out.println("nonce=" + byteArrayToHex(nonce));

    IvParameterSpec iv = new IvParameterSpec(nonce);
    SecretKey secret = new SecretKeySpec(key, "AES"); 
    Cipher cipher = Cipher.getInstance("AES/CTR/PKCS5PADDING");
    cipher.init(Cipher.DECRYPT_MODE, secret, iv);
    CipherInputStream cis = new CipherInputStream(fis, cipher);
    System.out.println(new String(cis.readAllBytes()));
}

public static String byteArrayToHex(byte[] a) {
    StringBuilder sb = new StringBuilder(a.length * 2);
    for (byte b: a)
       sb.append(String.format("%02x", b));
    return sb.toString();
}

It outputs key and IV, so you can compare with openssl output. I was using openssl 1.1.1

I am using 2.6.5 on my Mac, and there is no option: `-pbkdf2 ` — Norbert Koch, Sep 20 '19 at 11:12
Oh man, sorry for that, that's right. Sorry for delay, I'm in contact with a partner, and he is not answering for my requirement to change the command for encrypt a file. And on the Mac I'm currently only able to encrypt with LibreSSL. I will com back to you, after I have more information. I test it now in a docker container, to investigate the correct behavior. — Norbert Koch, Sep 23 '19 at 05:39

Java Decryption of openssl aes-256-ctr encrypted file

1 Answers1