583337 - Firefox detects, won't work with, server doing SSL DHE cipher suites with tiny keys

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Sean Martell]

http://www.enbridge.com/mypage

The above link caused a malformed SSL handshake in both Minefield nightly and Beta 2.  The page loaded properly in Beta 1 and 3.6.8

Comment 1

[Mike Beltzner [:beltzner, not reading bugmail]]

https://bugzilla.mozilla.org/buglist.cgi?resolution=FIXED,VERIFIED&chfieldto=2010-07-20%2015%3A00&chfield=resolution&query_format=advanced&chfieldfrom=2010-06-30%2011%3A00&product=Core&product=Firefox&product=NSPR&product=NSS&product=Toolkit is the beta1-beta2 regression range - we need to narrow that down.

Comment 2

[:Gavin Sharp [email: [e-mail]]]

Regression range:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=42249b1f40b3&tochange=6b58d2615145

I suspect: bug 575620.

Comment 3

[:Gavin Sharp [email: [e-mail]]]

Direct link to the problematic server: https://portal-plumprod.cgc.enbridge.com

Comment 5

[Mike Beltzner [:beltzner, not reading bugmail]]

Need a fix for this regression on 1.9.2 since we took bug 575620 on that branch. Either that or a backout of bug 575620 on branch.

(Martell: you just saved us a regression on a security update - way to go!)

Comment 6

[Wan-Teh Chang]

Wait!  That server proposes to the browser that 256-bit (32-byte) Diffie-Hellman
keys be used.  That key size is too small to be secure.  That's why NSS 3.12.7
aborts the SSL handshake.

The new Diffie-Hellman key size check is here:
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/nss/lib/ssl/ssl3con.c&rev=1.142&mark=5296,5305-5306#5296

Users can communicate with this website securely by turning off all ephemeral
Diffie-Hellman (DHE) SSL cipher suites as follows:
1. Type "about:config" in the location bar.
2. In the "Filter" field, type "ssl3.dhe".
3. Change the values of all the ssl3.dhe cipher suites to "false".

Comment 7

[Mike Beltzner [:beltzner, not reading bugmail]]

Are other servers likely to do that same thing? The way we discovered this was that someone was trying to see their gas bill, and Firefox stopped letting them do that. :/

Comment 8

[Wan-Teh Chang]

Attachment: Workaround patch for mozilla-1.9.2

It seems fine for a security update to become more strict
in checking cryptographic key sizes.  The issue is that
Firefox can communicate securely with this server if
it does not offer the ephemeral Diffie-Hellman (DHE)
SSL cipher suites.

After some experiments, I found a solution that I think
should satisfy everyone.  Please use this patch for
the mozilla-1.9.2 security update only.  For mozilla-2.0,
please insist that such servers are broken.

Details: I found that the server picks the DHE cipher suite
TLS_DHE_RSA_WITH_AES_256_CBC_SHA when Firefox uses unpatched
NSS.  If I apply this NSS patch, the server picks the
non-DHE cipher suite TLS_RSA_WITH_AES_256_CBC_SHA.
This patch changes NSS to list TLS_DHE_RSA_WITH_AES_256_CBC_SHA
after TLS_RSA_WITH_AES_256_CBC_SHA in its SSL ClientHello
message, thus affecting the server's selection (most servers
simply picks the first cipher suite it supports from the
client's list).

Comment 9

[Nelson Bolyard (seldom reads bugmail)]

That server is absolutely behaving insecurely.  Firefox is right to not
deceive its user into thinking that it has securely delivered the page 
to them.  The server is wrong and its admin should fix it, just the same
as with any other server misconfiguration.  

I believe we should not modify NSS to weaken Firefox's security for this.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

I remember this same problem happened numerous times years ago when we 
imposed minimum lengths on RSA modulus and public exponent sizes.  We 
began to get reports of servers that stopped working.  In one case, we 
found a server whose public key exponent was 1, which means that SSL to
their server wasn't doing any RSA encryption AT ALL.  Yet their admin 
wanted us to "fix the browser".  Instead, we did what was in the best 
interest of our users and their security.  I see no reason this case 
should be any different.

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 461710 [details] [diff] [review]
Workaround patch for mozilla-1.9.2

Wan-Teh, your patch will have the effect that, all over the internet, 
servers everywhere will stop selecting that cipher suite.  Most servers
who use it do so correctly.  In certain countries where the state has 
the right to demand that server operators give up their private keys,
DHE is the only thing that protects the users from retroactive decryption
of their prior SSL sessions by the state.  IMO, it is GROSSLY wrong of us
to deprive all Firefox users everywhere of this DHE cipher suite for the 
sake of a few users of one misconfigured server.  

So, r- for this patch for the NSS source trees in CVS.  

Now, Mozilla's HG copy of NSS is a "Downstream" copy, and they're free to 
do what they want with it.  If they think it's best to take this DHE cipher
suite away from the world to benefit a few users of some gas utility, then
so be it.

Comment 12

[Wan-Teh Chang]

Nelson, you are very right.  The difference this time
is that it *is* possible to communicate securely with
this server.  A server that prefers a DHE cipher suite
should look for one in the ClientHello's cipher suite
list, rather than just selecting the first supported
cipher suite from the list.

I proposed this patch for Mozilla's HG copy of NSS for
Firefox 3.6.x only.

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

Server admins in the U.K. _could_ disable all but the DHE cipher suites, 
I suppose.  But unless they do that, with this patch, instead of getting 
a weaker DHE cipher suite, they'd get TLS_RSA_WITH_AES_256_CBC_SHA, which 
is the non-DHE version of the same strong cipher suite they'd get without 
this patch.

Comment 14

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 461710 [details] [diff] [review]
Workaround patch for mozilla-1.9.2

Nelson: I'm not sure I understand your latest comments. Do you now concur with wtc that we should take this to address the compatibility regression on our stable shipping 1.9.2 branch?

It seems to me like the reordering only applies to the specific server cipher suite configuration we encountered here. What are the odds that the newly introduced DHE checks will affect other similar (but not necessarily identical) configurations?

Comment 15

[:Gavin Sharp [email: [e-mail]]]

Another alternative would be to introduce an NSS compile-time option to disable the additional DHE checks, which we can use on the 1.9.2 branch. I'm not very familiar with the severity of the problems the checks are supposed to prevent, though, so I don't know whether you would find that acceptable.

Other browsers (Chrome/Safari) seem to deal with this server just fine. Are they insecure?

It's somewhat difficult to take the lead on making sites incompatible for security reasons, since it gives users an incentive to switch to those other browsers (where the site they want to use works just fine). If we decide that we do want to take this hard line stance, we need to at the very least attempt to coordinate with other browsers.

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

No, Gavin, I do not concur with the patch as given.  
The chance that this patch will affect ALL users of DHE are 100%.

One site's misconfiguration doesn't constitute a crisis for Mozilla.
Defeating DHE security for users the world over just might!

Be clear, the proposal is to hurt the security of all FF users for the sake
of a few users who use a single misconfigured server.

Comment 17

[Wan-Teh Chang]

Attachment: Cipher suite list of IE 8.0 on Windows 7

gavin: even I will say no to disabling the additional DHE checks.
The current minimum of 512 bits is already very conservative.

This attachment shows that IE 8.0 on Windows 7 supports only
DHE cipher suites that require certificates with DSA keys
(very rarely used), and lists these DHE cipher suites at the
end.  This is why in practice no servers will select a DHE
cipher suite with IE.

Comment 19

[Wan-Teh Chang]

I suggest that Mozilla ask portal-plumprod.cgc.enbridge.com to fix
this server configuration problem.  The simplest fix is probably to
disable all DHE cipher suites.

It is bad to implement a workaround for just a single server.  We can
consider the workaround patch (attachment 461710 [details] [diff] [review])
if many servers are broken this way.

It would also be nice to add a new NSS SSL error code for this error
condition.  Right now NSS reports "a malformed ServerKeyExchange
message", which doesn't say the server's key is too weak.

Comment 20

[Nelson Bolyard (seldom reads bugmail)]

> It would also be nice to add a new NSS SSL error code for this error
> condition.  Right now NSS reports "a malformed ServerKeyExchange
> message", which doesn't say the server's key is too weak.

Agreed.  We should do that for all of our "server key too weak" detections.

Comment 21

[Boris Zbarsky [:bzbarsky]]

> I suggest that Mozilla ask portal-plumprod.cgc.enbridge.com 

In that case we need a TE bug on that, right?  Blocking the next branch release?

Comment 22

[Wan-Teh Chang]

bz: I filed TE bug 583914.  Could you please set the appropriate blocking flags
on the bug?  Thanks!

Comment 23

[LegNeato]

Ok, clearing the blocking flag for the branches

Comment 24

[:Gavin Sharp [email: [e-mail]]]

If other browsers give all DHE cipher suites lower priority (or don't support them at all), why can't we do the same?

This is just a problem with one site for now, but we haven't even released a beta with this change in it yet, so there's plenty of potential for more bustage to be discovered - particularly since we seem to be the only browser that behaves this way.

Comment 25

[noel.mulryan]

I am having issues with other sites too e.g. https://www.tescodiets.com. Every other browser is working for this website except the Firefox 4 beta.

Firefox seems to be aborting the connection with the alert "Illegal parameter", when the connection is viewed using wireshark, even though the key exchange data being passed is valid. The cipher that is being selected is the "TLS_DHE_RSA_WITH_AES_256_CBC_SHA" cipher as above.

I suspect there may be many other websites out there that will experience this issue too. Is this definitely something that will not be addressed?

Comment 26

[Nelson Bolyard (seldom reads bugmail)]

In reply to comment 25,
This is something that will be addressed by server administrators fixing 
their web sites to deliver real security rather than pretend security based
on tiny keys.  

In reply to comment 15,
Gavin, There is a mailing list (owned by Opera) on which representatives of 
browsers and crypto libraries discuss issues such as this.  It would be great
if MoCo had an active participant in that list.  But alas...

Comment 27

[:Gavin Sharp [email: [e-mail]]]

I'm not happy with WONTFIX, given the compatibility impact. Rejecting the weak ciphers is fine, but if we're going to do that we need to match other browsers in giving them the lowest priority to reduce the compatibility impact.

Comment 28

[:Gavin Sharp [email: [e-mail]]]

Just to be clear, my understanding is this:

1) NSS 3.12.7 rejects a certain class of "DHE" cipher suites (because they are insecure) that previous NSS versions didn't
2) NSS 3.12.7 lists these cipher suites first in its SSL ClientHello message, which means that servers are more likely to pick them

I am OK with 1). But I don't think it should have been done without changing 2) to match other browsers, because the result is that we are now apparently the only browser that will refuse to load pages that we used to load securely, and that are still possible to load securely.

Comment 29

[Boris Zbarsky [:bzbarsky]]

More precisely, we used to load them insecurely, since older NSS also had the DHE cipher suites early in the list.  But yes, it is still possible to load them securely.

Comment 30

[Wan-Teh Chang]

https://www.tescodiets.com also picks the cipher suite
TLS_DHE_RSA_WITH_AES_256_CBC_SHA and uses a 256-bit
(32-byte) ephemeral Diffie-Hellman key.  The generator
element (DH_g) is 2 for both servers.

https://www.tescodiets.com identifies itself as:
    Server: Microsoft-IIS/7.5
    X-AspNet-Version: 2.0.50727
    X-Powered-By: ASP.NET

https://portal-plumprod.cgc.enbridge.com identifies itself as:
    Server: Apache

My workaround also allows Firefox to connect to
https://www.tescodiets.com/.

Comment 31

[Wan-Teh Chang]

(In reply to comment #25)
noel.mulryan: given your tescodiets.com email address,
you're in a perfect position to have the administrator
of your company's website fix the server configuration
error.  Bug 583914 has the instructions for the server
administrator.

I'd appreciate if you could list the other websites
that you've encounter this problem with.  This will help
us understand if the problem with these websites comes
from a common piece of software.  Thanks!

Comment 32

[Nelson Bolyard (seldom reads bugmail)]

Opera detects these weak keys, and refuses to use them.  
Instead, it disables DHE for that server only and renegotiates.
That's secure, and doesn't deprive any one of correct use of DHE.

It's also a PSM feature, not an NSS feature.
If you want it, reassign this bug to PSM.

Comment 33

[Mike Hommey [:glandium]]

Another impacted site: https://www.citylink.com.au

Comment 34

[Wan-Teh Chang]

Thanks.  https://www.citylink.com.au behaves exactly
like the other two problematic servers.  It doesn't
send a Server response header.

Comment 35

[LegNeato]

Note that this bug should likely be blocking 2.0 final+ as 584138 was blocking it and is now invalid.

Comment 36

[Boris Zbarsky [:bzbarsky]]

It should certainly block.  The question is what.  I feel we need to at least ship a beta with any changes we make here.  Beta5 is probably the right milestone to block, given the feature freeze aspect of things...

Comment 37

[Mike Hommey [:glandium]]

FWIW, please note the NSS change also impacts chromium/google chrome.

Comment 38

[Nelson Bolyard (seldom reads bugmail)]

Chrome's developers support the NSS change.

Comment 39

[Wan-Teh Chang]

(In reply to comment #35)
Christian: I am not familiar with how Mozilla does technical
evangelism.

In comment 22, I filed TE bug 583914 as bz suggested, and
closed this bug WONTFIX.  Then gavin reopened this bug in
comment 27.  My TE bug can't block a release, but the dummy
shadow bug 584138 got closed as invalid.

It seems that nothing will be done soon about my TE bug
583914 because it doesn't block any release.

Comment 40

[LegNeato]

Right, I don't expect the TE bug 583914 to be fixed or to block. It is my understanding this bug has engineering work that can be done from comment 32.

We would like to get a patch for this so that we can ship it in 3.6.9. From a FF user's standpoint, this bug will be a regression once we ship next month (because we upgraded to this behavior, bug 575620).

I would rather not revert back to an older version of NSS, but code freeze is tomorrow. I would be willing to push the freeze back for this bug if it gets some traction in the next day. Please let me know if this is not going anywhere and instead I should roll back nss.

Comment 41

[Wan-Teh Chang]

Attachment: Workaround patch for mozilla-1.9.2, v2

For mozilla-1.9.2 only, I propose this patch.

To review the NSS patch, verify that the two arrays of
SSL cipher suites in ssl3con.c and sslenum.c are kept
in the same order, as this comment notes:
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/nss/lib/ssl/ssl3con.c&rev=1.143&mark=103-104#101

Comment 42

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

I guess this is fine, though Nelson seemed to have some objections to this....  Were they weak enough that shipping a security release (or possibly all future 3.6.x releases) with this patch is ok?

Comment 43

[Nelson Bolyard (seldom reads bugmail)]

I object to this because it takes DHE away from ALL Firefox users for the 
sake of few users who visit broken servers.  

A better fix is to enhance PSM's TLS intolerance fallback to also disable 
the DHE cipher suites.  That's a very small change to PSM.

I won't block this change if it is made ONLY in Mozilla's downstream Hg repository.  I don't want to see it in CVS.  SR- for the CVS tree!

Comment 44

[LegNeato]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

a=LegNeato for 1.9.2.9. Please land this workaround on mozilla-1.9.2 default.

Comment 45

[Wan-Teh Chang]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

Pushed to mozilla-1.9.2 default:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/68a4d8f0a18f

Comment 46

[Michael Lefevre]

This was a Firefox 4 blocker as well, and it's not fixed there as far as I can see, only on the branch. So if this bug is getting marked fixed, I guess a new bug should be filed for doing the "better fix" for Firefox 4?

Comment 47

[Wan-Teh Chang]

For Firefox 4 I won't work around the broken servers.
I filed TE bugs for the broken servers, which are
listed in the "Depends on" field.  I propose that
those TE bugs be Firefox 4 blockers.

Comment 48

[Boris Zbarsky [:bzbarsky]]

We still need a bug for the better PSM fallback, right?

Comment 49

[Mike Beltzner [:beltzner, not reading bugmail]]

(In reply to comment #47)
> For Firefox 4 I won't work around the broken servers.
> I filed TE bugs for the broken servers, which are
> listed in the "Depends on" field.  I propose that
> those TE bugs be Firefox 4 blockers.

I'm not sure that's right. Doesn't the previous discussion indicate that the broken servers are likely to be a common pattern, and we should instead interpret the safest possible action?

I agree that we should get a Firefox 4 bug on file to ensure we track this.

Comment 50

[:Gavin Sharp [email: [e-mail]]]

I filed bug 587407 on PSM.

Comment 51

[Al Billings [:abillings - ex-MoCo]]

What is the STR here? I note that http://www.enbridge.com/mypage loads without any errors in the nightly 1.9.2.9pre build and redirects to https://portal-plumprod.cgc.enbridge.com/portal/server.pt?space=CommunityPage&control=SetCommunity&cached=true&CommunityID=203&PageID=0 with no warning messages. 

I assume that there is probably more to test here or is there?

This was tested using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.9pre) Gecko/20100817 Namoroka/3.6.9pre ( .NET CLR 3.5.30729).

Comment 52

[Wan-Teh Chang]

Al: the STR is:

1. Load http://www.enbridge.com/mypage in Firefox 4 Beta 3.
You should get a "Secure Connection Failed" error page with
the error code: ssl_error_rx_malformed_server_key_exch

2. Now load http://www.enbridge.com/mypage in the
nightly 1.9.2.9pre build.  You should not get this error page.

Comment 53

[Al Billings [:abillings - ex-MoCo]]

Yes, I already did that. I wanted to be sure that there is nothing more to do here.

Marking verified for 1.9.2.

Comment 54

[Wan-Teh Chang]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

Firefox 3.5.12 also needs this workaround.

Comment 55

[Al Billings [:abillings - ex-MoCo]]

You missed the train for 3.5.12. We built today and are going to beta on Thursday barring rebuilds.

Comment 56

[LegNeato]

If we need to spin again for a regression I'll want this in, otherwise it'll get in .13

Comment 57

[Boris Zbarsky [:bzbarsky]]

Why does 3.5 need this?  Bug 575620 didn't land on 1.9.1, did it?

Comment 58

[Wan-Teh Chang]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

bz: Firefox 3.5 doesn't check ephemeral Diffie-Hellman key
sizes.  This patch prevents Firefox 3.5 from using ephemeral
Diffie-Hellman cipher suites with the broken servers.

Ideally we should also backport the key size checks:
http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=ssl3con.c&branch=&root=/cvsroot&subdir=mozilla/security/nss/lib/ssl&command=DIFF_FRAMESET&rev1=1.137&rev2=1.138

But this patch alone is good enough.

Comment 59

[:Gavin Sharp [email: [e-mail]]]

It's a bit confusing to be proposing that we take the patch on 1.9.1 in this bug - it's "necessary to ensure security" (because 1.9.1 doesn't check key sizes), which is different than the 1.9.2 patch, which is "necessary to avoid breaking sites due to extra security checks." But I agree that we should take it - perhaps file a new 1.9.1-only bug?

Comment 60

[Boris Zbarsky [:bzbarsky]]

Yeah, it sounds like the 1.9.1 patch is fixing a different bug and should be tracked separately.

Comment 61

[LegNeato]

Yeah, talking with abillings today we realized this particular bug can't happen on 1.9.1 as it was a behavior change with the newer nspr, which didn't land on 1.9.1.

Comment 62

[Wan-Teh Chang]

Sigh. One needs a strong will to persevere in this environment.

Fact: 1.9.1 accepts tiny 256-bit Diffie-Hellman keys of the broken servers.

Fact: 1.9.1 has been doing that since day one, so there was no behavior change.

If you think this is worth fixing, I will follow the proper process and open a
new 1.9.1-only bug.

Comment 63

[:Gavin Sharp [email: [e-mail]]]

Yes, it is worth fixing. Please do file the new bug (and CC me?).

Comment 65

[Daniel Veditz [:dveditz]]

We'll need this on the 1.9.1 branch since we'd like to upgrade to nss 3.12.8 there.

Comment 66

[Daniel Veditz [:dveditz]]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

Approved for 1.9.1.14, a=dveditz for release-drivers (to be landed after upgrading to NSS 3.12.8)

Comment 67

[Wan-Teh Chang]

Comment on attachment 465229 [details] [diff] [review]
Workaround patch for mozilla-1.9.2, v2

Pushed to mozilla-1.9.1 in changeset 8ee042940966:
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/8ee042940966

Comment 68

[Nick Lowe]

Please make sure that this is seen as a kludge and a quick compatibility hack. This bug is definitely not fixed and the patch committed introduces behaviour that isn't desirable in the slightest and is regressive. It degrades security for Firefox users to workaround broken servers.

Can this bug be reopened, therefore?

The correct compatibility approach is to renegotiate on failure, precluding the use of a DHE based cipher suite on the second attempt.

This as said in comment 32:
https://bugzilla.mozilla.org/show_bug.cgi?id=583337#c32

DHE should be used in preference as it prevents retrospective decryption of intercepted and captured encrypted traffic. This decision, therefore, tangibly weakens the security of Firefox. That's bad.

This was also said in comment 11:
https://bugzilla.mozilla.org/show_bug.cgi?id=583337#c11

The fact that other browsers do, or might, not get this right does not make this the correct implementation decision. That's merely blind deference to the appeal to authority fallacy in any case; it isn't based in evidence or reason. Firefox should hold itself to a higher standard and lead the way, not tag along.

The same arguments were made back in the day against introducing the use of TLS extensions as many servers broke with it. Finally, Microsoft forced the issue with the advent of IE7 and Vista and the world moved on and fixed its servers.

http://blogs.msdn.com/b/wndp/archive/2006/04/12/tls-enabled-by-default.aspx
http://blogs.msdn.com/b/ie/archive/2005/10/22/483795.aspx

In abstract, I would certainly venture that security is really the wrong thing to be compromising against in such a way for compatibility purposes.

That's moot in this case anyway as there is a correct approach here, yet it hasn't been implemented!?...

Cheers,

Nick

Comment 69

[Robert Relyea]

It's unlikely we are going to actually change FF 3.6's PSM. The discussion on the pros and cons of the different fix options are in bug 587407 .