TimeOut exception when publishing or updating content

Manjeera_Reddy · April 15, 2020, 8:46am

I have…

[ ] Checked the logs and have provided the logs if I found something suspicious there

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[ x] Bug report
[ ] Performance issue
[ ] Documentation issue or request

Current behavior

We are connecting to Azure Webhook on publishing or on updating the content.
After we publish/update content, we verified the logs under “Rules”. If the exection time is crossing 2sec then we see “Action Timed Out” exception even though connecting to Azure and executing the logic is successful.
Because of this “Action Timed Out” exception,Squidex is trying to enqueue again the same request.

Logs
Response:

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
—> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
—> System.Net.Sockets.SocketException (125): Operation canceled
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask1 task, Int32 min, Int32 initial) at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory1 buffer)
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
— End of inner exception stack trace —
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Squidex.Extensions.Actions.HttpHelper.OneWayRequestAsync(HttpClient client, HttpRequestMessage request, String requestBody, CancellationToken ct) in /src/extensions/Squidex.Extensions/Actions/HttpHelper.cs:line 46

Action timed out.

Elapsed 00:00:02.0020000.

Expected behavior

This Azure Webhook should not show timedout if the execution is successful.

Minimal reproduction of the problem

Environment

[ ] Self hosted with docker
[ ] Self hosted with IIS
[ ] Self hosted with other version
[ ] Cloud version

Version: [VERSION]

Browser:

[x ] Chrome (desktop)
[ ] Chrome (Android)
[ ] Chrome (iOS)
[ ] Firefox
[ ] Safari (desktop)
[ ] Safari (iOS)
[ ] IE
[ ] Edge

Others:

Sebastian · April 15, 2020, 8:47am

I am not sure how I can help you here. But I think you are using an older version, because the timeout of 2 seconds has been increased.

Either your webhook is not reachable or it just takes too long.

Btw: Can you please format your log with code blocks?

Manjeera_Reddy · April 16, 2020, 12:37pm

Hi Sebastian,
Thanks for the response.

I created azure function and hooked to squidex. I see the execution time is 6 to 8 sec.
Because of this I am getting timeout exception even though executing azure function is success.
So the rule is failed Squidex is trying to trigger the request multiple times.
Is it possible to stop the rule to get executed multiple times by doing any configuration change?

Thanks,
Manjeera T

Sebastian · April 16, 2020, 12:45pm

You can only increase the timeout.

Manjeera_Reddy · April 16, 2020, 10:02pm

Hi Sebastian,
Thanks for the response.
Is it possible to increase the timeout in squidex via configuration.

Thanks,
Manjeera T

Sebastian · April 17, 2020, 10:26am

This is the current timeout: But I have no idea which version you are using.

github.com

Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L56


  /*
   * Use custom langauges where the key is the language code and the value is the english name.
   */
  "custom": ""
},


"rules": {
  /*
   * The timeout to execute rule actions.
   */
  "executionTimeoutInSeconds": 10
},


"ui": {
  /*
   * Regex suggestions for the UI
   */
  "regexSuggestions": {
    // Regex for emails.
    "Email": "^[a-zA-Z0-9.!#$%&’*+\\/=?^_`{|}~-]+@[a-zA-Z0-9-]+(?:.[a-zA-Z0-9-]+)*$",
    // Regex for phone numbers.

Manjeera_Reddy · April 18, 2020, 4:43am

Hi Sebastian,
I am using Squidex with Free Plan. I tried to check the version of squidex but not able to find it.
Can you please let me know where can i check it.

Thanks,
Manjeera T

Sebastian · April 18, 2020, 8:01am

When you use the cloud there is no way to do it. But I see if can increase the timeout

Manjeera_Reddy · April 22, 2020, 8:10am

Hi Sebastian,
We are using cloud free version.

Response timeout is provided below

{"type":"ProductUpdated","payload":{"$type":"EnrichedContentEvent","type":"Updated","id":"2e8c1868-ab1f-4989-8742-b5d7ff8e37c7","created":"2020-04-14T12:17:47Z","lastModified":"2020-04-22T08:07:50Z","createdBy":"subject:5df1c9e89997500001464863","lastModifiedBy":"subject:5df1c9e89997500001464863","data":{"name":{"iv":"Rad to the Bone Slim Fit Tee"},"code":{"iv":"96658-19VM"},"variants":{"iv":[]},"style":{"iv":[]},"description":{"iv":"96658-19VM"},"startprice":{"iv":"555"},"quantity":{"iv":"2"},"listingduration":{"iv":"Days_7"},"listingtype":{"iv":"FixedPriceItem"},"paymentmethods":{"iv":"PayPal"},"paypalemailaddress":{"iv":"phanikumar_rentachintala-facilitator@live.com"},"postalcode":{"iv":"3002"},"pictureurl":{"iv":"https://rukminim1.flixcart.com/image/880/1056/k65d18w0/shirt/p/4/t/48-bfrybluesht02ab-being-fab-original-imaecvnx68fyvkjp.jpeg?q=50"},"returnsacceptedoption":{"iv":"ReturnsAccepted"},"refundoption":{"iv":"MoneyBack"},"returnswithinoption":{"iv":"Days_30"},"shippingcostpaidbyoption":{"iv":"Buyer"},"shippingtype":{"iv":"Flat"},"shippingservicepriority":{"iv":"1"},"shippingservice":{"iv":"AU_ExpressDelivery"},"shippingservicecost":{"iv":"22"},"shippingserviceadditionalcost":{"iv":"44"},"suggestion":{"iv":"Short Sleeve Tees"},"isvariant":{"iv":true},"sizevariant":{"iv":["6cb53d62-96e5-4d4d-adc5-46b3dd069cc2"]},"itemspecific":{"iv":["358c3621-bf04-41ed-83a8-d328e3fed3e7","39b72595-4d76-4448-8d35-b945c2dc436e","cf077948-a23e-4f33-b8b6-3b3a3db03dc2","cec8ee8f-d4e3-4075-ba3a-d29207790604"]}},"dataOld":{"name":{"iv":"Rad to the Bone Slim Fit Tee1"},"code":{"iv":"96658-19VM"},"variants":{"iv":[]},"style":{"iv":[]},"description":{"iv":"96658-19VM"},"startprice":{"iv":"555"},"quantity":{"iv":"2"},"listingduration":{"iv":"Days_7"},"listingtype":{"iv":"FixedPriceItem"},"paymentmethods":{"iv":"PayPal"},"paypalemailaddress":{"iv":"phanikumar_rentachintala-facilitator@live.com"},"postalcode":{"iv":"3002"},"pictureurl":{"iv":"https://rukminim1.flixcart.com/image/880/1056/k65d18w0/shirt/p/4/t/48-bfrybluesht02ab-being-fab-original-imaecvnx68fyvkjp.jpeg?q=50"},"returnsacceptedoption":{"iv":"ReturnsAccepted"},"refundoption":{"iv":"MoneyBack"},"returnswithinoption":{"iv":"Days_30"},"shippingcostpaidbyoption":{"iv":"Buyer"},"shippingtype":{"iv":"Flat"},"shippingservicepriority":{"iv":"1"},"shippingservice":{"iv":"AU_ExpressDelivery"},"shippingservicecost":{"iv":"22"},"shippingserviceadditionalcost":{"iv":"44"},"suggestion":{"iv":"Short Sleeve Tees"},"isvariant":{"iv":true},"sizevariant":{"iv":["6cb53d62-96e5-4d4d-adc5-46b3dd069cc2"]},"itemspecific":{"iv":["358c3621-bf04-41ed-83a8-d328e3fed3e7","39b72595-4d76-4448-8d35-b945c2dc436e","cf077948-a23e-4f33-b8b6-3b3a3db03dc2","cec8ee8f-d4e3-4075-ba3a-d29207790604"]}},"status":"Published","partition":2005368591,"schemaId":"1e9da355-c6c0-4fa8-953c-5bf106735211,product","actor":"subject:5df1c9e89997500001464863","appId":"366e131f-7265-40e9-8a73-f208922d1ce9,harley-davidson-australia","timestamp":"2020-04-22T08:07:50Z","name":"ProductUpdated","version":8},"timestamp":"2020-04-22T08:07:50Z"}

Response:

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Squidex.Extensions.Actions.HttpHelper.OneWayRequestAsync(HttpClient client, HttpRequestMessage request, String requestBody, CancellationToken ct) in /src/extensions/Squidex.Extensions/Actions/HttpHelper.cs:line 46

Action timed out.

Elapsed 00:00:02.0050000.

Thanks,
Manjeera T

Sebastian · April 22, 2020, 8:15am

I have not updated the timeout yet, it was indeed set to 2 seconds. But please keep in mind that the timeout is needed to guarantee that the servers keep stable. If oyu have long running operation I would push your events to azure queue.

Manjeera_Reddy · April 25, 2020, 4:55am

Hi Sebastian,
I have along running operation which will take around 6 to 12 sec.

Thanks,
Manjeera T

Sebastian · April 25, 2020, 8:35am

Then i recommend azure queue in your case