EMERGENCY website down

Support
1

I have…

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x ] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Expected behavior

Minimal reproduction of the problem

Environment

App Name:

  • [ ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: [VERSION]

Browser:

  • [x ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

All our calls to their API are returning this error now. He says no changes on their side could have caused it, but as we know we haven’t touched the code on our side in months if not over a year.

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

// Server Error in ‘/’ Application.

// Error parsing undefined value. Path ‘’, line 1, position 1.

//Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

//Exception Details:** Newtonsoft.Json.JsonReaderException: Error parsing undefined value. Path ‘’, line 1, position 1.

//Source Error:

//Line 155: public static Home FromJson(string json)
//Line 156: {
//Line 157: return JsonConvert.DeserializeObject(json, //QuickType.Home.Converter.Settings);
//Line 158: }
//Line 159: }`

//Source File: d:\inetpub\wwwrootNEW\App_Code\Home.cs Line: 157

//Stack Trace:

[JsonReaderException: Error parsing undefined value. Path '', line 1, position 1.] //Newtonsoft.Json.JsonTextReader.ParseUndefined() +308 //Newtonsoft.Json.JsonTextReader.ParseValue() +1052 //Newtonsoft.Json.JsonTextReader.Read() +152 //Newtonsoft.Json.JsonReader.ReadAndMoveToContent() +21 //Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) +97 //Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) +764 //Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) +336 //Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) +279 //Newtonsoft.Json.JsonConvert.DeserializeObject(String value, JsonSerializerSettings settings) +57 //QuickType.Home.Home.FromJson(String json) in d:\inetpub\wwwrootNEW\App_Code\Home.cs:157 //CmsHelper.getHomePage(String lang) in d:\inetpub\wwwrootNEW\App_Code\CmsHelper.cs:607 //CTW_demo2_Default2.Page_Load(Object sender, EventArgs e) +309 //System.Web.UI.Control.LoadRecursive() +71 //System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3178

//Version Information Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.34237

2

Sorry, but I cannot work with this stack trace. What are you doing there and what do you expect? You have to narrow it down. Usually, when you deserialize from JSON you have a object model, that is also need as a reference to understand what the issue might have been.

In the previous conversation you said you have issues with obtaining the token, but I don’t see it here.

3

As per David: The call to get the token, which has been working for years (aside from one outage last year which somehow “fixed itself”) has suddenly stopped working.

4

I have seen 4 different information that confuse me:

  1. David has sent me an error report, that shows that authentication fails. In the sample from postman, the parameters like grant_type have been passed via query string, but the author of the authentication library has tole me that query strings are not supported and was never supported. Furthermore the code was not showing any query string logic. There was a concrete error message and the JSON was parsed properly.

  2. The code above shows an JSON error message but not enough information to narrow down the problem. I am not even sure that it has anything to do with obtaining the token and whether it has something to do with authentication.

  3. David has also sent me an message that was indicating that there was a timeout exception. But again I don’t know how often it has happened or where exactly. I have tested the token code several times in a console application and I had between 40-50ms for every single call. But I am closer to the data center of course.

  4. I was also told that the problem started this morning. But there was no deployment or something like that. The last deployment was this Wednesday: Oct 25, 2023, 5:23:11 PM (UTC + 2).

For me it is super confusing and I cannot help without further information. Provide full stack traces and the code that is affected and try to find out how often a problem actually happened.

5

The error that DotNet is throwing is:

System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

After doing some googling, it seems this may be the answer:

Basically looks like somebody changed some security settings, as I quote from that page:

You may get this error when trying to call an external API. This error is related to the Security Protocol Type, it is most likely caused by your application’s default security protocol type being set too low, A lot of external APIs now expect requests using TLS 1.2 or above.

So it seems that SquidEx now requires TLS1.2 or above, and that change must have been made quite recently (i.e. went into effect Friday).

Please advise as we’d like to understand the bigger context of this change to TLS1.2 (which I agree is a good idea) and how we can have these type of potentially breaking changes communicated in advance?

6

Hi,

thats true. TLS has been changed to 1.2, but it was 2 month ago. Therefore I have not thought about this:

image
image1332×535 19.3 KB

TLS 1.1 is not secure anymore, deprecated and will eventually be turned off anyway. So the question is: Is TLS 1.2 supported by your setup. If yes, then something is wrong on your side, for example if you create a HttpClient and keep it forever then you will run into these issues. The same issue can happen if I would have to make a DNS change, for example move the server to another location or so. Because even if I would keep the old and new server running for a week, the DNS would be cached on your side, as long as the application lives.

If TLS 1.2 is not supported I can revert my change (for a few months), but then I don’t understand why it happened now. Do you use long cache times?

7

Sebastian,

Something isn’t adding up here. The metrics clearly show this started impacting us on Friday only. And you can clearly see plenty of traffic before that, so something changed just this friday:
image

I agree that TLS1.2 is the better option. And we have turned on TLS1.2 are the site works now. However, TLS1.1 was clearly working for us up until last Friday. Was the change perhaps not pushed out to all your servers until Friday?

But for that matter, we did not receive any notification 2 months ago about the turning off of 1.1.

8

This would be a major bug in cloudflare. I don’t understand it.

Yes, this was a mistake because I thought it would be a no-brainer and nobody would be affected by that.

9

The site is down again.

10

Evan, this is because my solution was a temporary one. You need to enable TLS1.2 at the server level as we discussed. No need to continue this conversation here.

1 Like
11

Hi Sebastian,

We are in the process of server upgrades and the client has asked me to inquire if there is any way we can have TLS 1.1 turned back on temporarily just until we resolve the server issue. It’s a desperate situation as the website has been down since Friday and our current plan won’t have service restored for another 24 hours at minimum. Please sir, if we could trouble you for some special consideration here would go a long way. 3 days would be extremely appreciated.

12

Done, reverted to 1.1

13

THANK YOU SO MUCH SIR We really appreciate this!

14

Hi Sebastian, was a notification ever sent out regarding the change in security parameters? If there was, Mr Silny would like to know which email address was used and moving forward would also like notifications of this nature sent to his email. jsilny@jsilny.org

Thanks again for everything!

15

Have you received the emails about the new editor? If yes, you should be fine.

16

I have not received any email about the editor. After discussion here, we believe that all three of us should be on the notification list:

  1. dgtman@gmail.com
  2. jsilny@jsilny.org
  3. espector@iontech247.com
17

Hi Sebastian,

We are still working on the necessary upgrades for TLS1.2 is there any way we can have an additional week of TLS 1.1 support?

18

Sure, i am not in a hurry.

19

Sir, Thank you so much!

20

Hi Sebastian, it appears the site is down and I haven’t made any changes since the last time it was up. Is TLS 1.1 still enabled for us? If it not can you please turn it back on we are still working on making the necessary changes.

The trace information on the diagnostic page David made for indicates its the same problem with authentication failing because the party has closed the transport stream.

aspx.page Begin Load 0.000219 0.000009
BYteARray: 0.001421 0.001202
System.Byte[] 0.001456 0.000035
BYteARraySize :131 0.001467 0.000012
ERROR = System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.