Bug #4060
open
Encryption framework's bignum (and consumers) need to increase from 4096-bits.
0%
Description
A SmartOS user with an 8192-bit RSA key states that he couldn't log in, with the system saying:
sshd[16521]: [ID 800047 auth.error] error: RSA_public_decrypt failed: error:8006A076:PKCS#11 engine:PK11_GET_PUB_RSA_KEY:C_CreateObject failedI can reproduce this just by producing an 8192-bit key on a mac (ssh -t rsa -b 8192) and attempting to use it.
Updated by Rich Lowe over 9 years ago
This is a limit in the encryption framework, it refuses larger RSA keys larger than 4096 bits for reasons that aren't clear to me.
There's a vague rumour in some old notes that the reason somehow involves our bignum code, but there's nothing to really back this up.
Updated by Dan McDonald over 9 years ago
I found a constant in usr/src/common/bignum/bignum.h called BIGTMPSIZE, which is set to 65. bignum uses either 32 or 64-bit chunks. 64 * 64 4096.
It MAY (emphasis on MAY) be as simple as increasing BIGTMPSIZE to a larger number, perhaps 129 (8192 + extra chunk) or even 256 (16384 + extra chunk). One possible downside is that there are a LOT of stack variables:
osdev2(common/bignum)[0]% grep BIGTMPSIZE * */*
bignum.h:#define BIGTMPSIZE 65
bignumimpl.c: BIG_CHUNK_TYPE tmp1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tresultvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE bblowvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE bbhighvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE carry[BIGTMPSIZE];
bignumimpl.c: if ((big_cpu BIG_CPU_INTEL) && (nlen <= BIGTMPSIZE)) {
bignumimpl.c: BIG_CHUNK_TYPE rrvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE rrvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE mavalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE rrvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t3value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE prodvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t3value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE kivalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE ovalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE nminus1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE Lkminus1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE Lkvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t3value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t4value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t5value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t6value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t7value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t8value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
osdev2(common/bignum)[0]%
Anyone interested in testing this out?
Updated by Rich Lowe over 9 years ago
This is also SmartOS issue: https://github.com/joyent/smartos-live/issues/291
Updated by Dan McDonald over 6 years ago
With the deprecation of SunSSH, this issue as-written becomes obsolete. The underlying issue -- namely usr/src/crypto & friends can't cope with bignums over 4096-bit -- is still a problem, and perhaps still should be tracked by this bug? Any objections to a rename?
Updated by Dan McDonald over 6 years ago
- Subject changed from sshd doesn't accept large RSA keys to Encryption framework's bignum (and consumers) need to increase from 4096-bits.
- Difficulty changed from Medium to Hard