Project

General

Profile

Bug #4060

Encryption framework's bignum (and consumers) need to increase from 4096-bits.

Added by Rich Lowe over 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
lib - userland libraries
Start date:
2013-08-20
Due date:
% Done:

0%

Estimated time:
Difficulty:
Hard
Tags:
needs-triage
Gerrit CR:

Description

A SmartOS user with an 8192-bit RSA key states that he couldn't log in, with the system saying:

sshd[16521]: [ID 800047 auth.error] error: RSA_public_decrypt failed:       error:8006A076:PKCS#11 engine:PK11_GET_PUB_RSA_KEY:C_CreateObject       failed

I can reproduce this just by producing an 8192-bit key on a mac (ssh -t rsa -b 8192) and attempting to use it.

#1

Updated by Rich Lowe over 7 years ago

This is a limit in the encryption framework, it refuses larger RSA keys larger than 4096 bits for reasons that aren't clear to me.

There's a vague rumour in some old notes that the reason somehow involves our bignum code, but there's nothing to really back this up.

#2

Updated by Dan McDonald over 7 years ago

I found a constant in usr/src/common/bignum/bignum.h called BIGTMPSIZE, which is set to 65. bignum uses either 32 or 64-bit chunks. 64 * 64 4096.

It MAY (emphasis on MAY) be as simple as increasing BIGTMPSIZE to a larger number, perhaps 129 (8192 + extra chunk) or even 256 (16384 + extra chunk). One possible downside is that there are a LOT of stack variables:

osdev2(common/bignum)[0]% grep BIGTMPSIZE * */*
bignum.h:#define BIGTMPSIZE 65
bignumimpl.c: BIG_CHUNK_TYPE tmp1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tresultvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE bblowvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE bbhighvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE carry[BIGTMPSIZE];
bignumimpl.c: if ((big_cpu BIG_CPU_INTEL) && (nlen <= BIGTMPSIZE)) {
bignumimpl.c: BIG_CHUNK_TYPE rrvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE rrvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE mavalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE rrvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t3value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE prodvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t3value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE kivalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmp2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE ovalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE nminus1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE Lkminus1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE Lkvalue[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t1value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t2value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t3value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t4value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t5value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t6value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t7value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE t8value[BIGTMPSIZE];
bignumimpl.c: BIG_CHUNK_TYPE tmpvalue[BIGTMPSIZE];
osdev2(common/bignum)[0]%

Anyone interested in testing this out?

#3

Updated by Rich Lowe over 7 years ago

#4

Updated by Dan McDonald over 4 years ago

With the deprecation of SunSSH, this issue as-written becomes obsolete. The underlying issue -- namely usr/src/crypto & friends can't cope with bignums over 4096-bit -- is still a problem, and perhaps still should be tracked by this bug? Any objections to a rename?

#5

Updated by Dan McDonald over 4 years ago

  • Subject changed from sshd doesn't accept large RSA keys to Encryption framework's bignum (and consumers) need to increase from 4096-bits.
  • Difficulty changed from Medium to Hard

Also available in: Atom PDF