Hi @Sebastian,
We are getting this error for https://cloud.squidex.io/api/content/vannatest/graphql
429 Error: Our Website https://dev.vanna.io is down
Then you have to upgrade your plan.
Your plan has 500k calls included, after 1.500k it should actually block your requests, so I am not sure how you have been able to make 2000k.
Have you not received an email?
@Sebastian Thanks for the quick response
I have yet not taken this in production, barely two people are using the website discretely to test. I haven’t received any email, and I am not sure how these calls have sporadically increased.
I will look at my end, please look at your end as well as no email alerts have come. I will try to change keys
Yes, I am just about to download the backup of the database to reproduce it locally.
Are you using the CDN?
@Sebastian I don’t think so, we were planning to use the CDN but as per your last confirmation on one of the previous tickets (API performance), you highlighted our current App (vannatest) has not used CDN endpoints yet. I will check back with my frontend team and come back.
I have reset your usage data to solve the issuje for now and will dig into it. I can take a little bit until it works again.
I think it must be on my end. When you download the full logs, the amount does not match to the shown total.
Sorry, I was wrong. The download of the logs has stopped too early. It is actually huge and the range of the numbers match.
@Sebastian I can see the numbers rise very quickly again on the Dashboard for API calls. When I guess not even two users are testing the platform. It is some kind of automated API calls it seems
Could be, but it is coming from your client. Do you have load tests or API tests or so? I can see in the logs that all requests are done via the graphql endpoint.
What also looks suspicious is that you do a lot of duplicate requests on your site. I see getArticle twice, getStaticContent twice and so on.
You are always loading the content for EN and zh-TW
@Sebastian No tests have been implemented, In the logs we can only see all calls are via API GraphQL POST and hence don’t have further details to ascertain which queries are repeatedly being called so that we can diagnose which calls are the culprit. We are also running the mobile and desktop builds locally but don’t have any repeated calls.
A few more things:
-
You have to notice that each requests has a cost associated. Some API calls only costs 0.1 calls, and graphql costs 2 calls, because you can do very heavy requests with GraphQL.
-
There is a batch endpoint for graphql: https://www.apollographql.com/blog/batching-client-graphql-queries-a685f5bcd41b/ The endpoint for batching ends with /graphql/batch instead of /graphql
-
You can save API calls with a CDN. For example you could use cloudflare as an edge server and route all requests through and then define a short cache duration for some of the API calls. But you have to move to GET requests then.
I don’t think that there is something fundamentally wrong with the request logs.
@Sebastian Humble request !!! Is it possible to get a dump of detailed logs(beyond the ones I downloaded from the Squidex Dashboard) for like last 2-3 days for our Cloud Instance for API calls that are made
e.g.
What queries are being made for each POST request. That will help us internally narrow down the API calls. Basically the PAYLOAD of each query for last couple of days
I understand your recommendations on COST per GraphQL API Calls and the Caching Optimisations. I am more concerned around the REPEATED CALLS for LOADING DATA via POST requests (which creates a feeling of AUTOMATED TESTs) as the underlying problem that needs to fixed to reduce majority of the bulk calls.
Hi, the problem is that I cannot log the POST payloads. It is just too much data, I do not have it available.
I can extend the logging to store query strings as well, then you will have the information when you use GET instead of POST.
If you create a new client and test it for a few hours it is is very unlikely that somebody else is using it. Then you can log it on your side. For example, you can use google analytics for that.
I have been monitoring the calls and we reduced couple of our other instances using Squidex API just to see the API Usage the outcome was on one day the calls overshot the previous day and the next day it subsided.
I am sure now there is another reason beyond our reach why these usage statistics of this high order is happening.
Based on your recommendation I am going to try and replace the API keys in our builds just to ensure unauthorised usage is not happening when it comes to stress testing through some unknown actor
@Sebastian it seems something erroneous(funny) has started again for API calls usage since yesterday. It seems to consuming the API count and has already crossed our monthly quota in 1.5 day. PFA the screenshot here and logs also. I am going to make key replacement as an interim fix but I am not sure what is the issue here.
API QUOTA BURNOUT in 1.5 DAYS like 500k Calls
LOGS
https://drive.google.com/file/d/1-aOqcvKYpMBuLnO7rs9j06NYwlb1DFax/view?usp=sharing
I have reset your usage data and will have a look now.
@Sebastian no relief on this front. It seems to be there are some retry calls that keep happening as a result of which the number of calls never subside. This issue is still unresolved.
Also we recently got charged for API calls but this somehow is undetectable why such heavy number of calls are being made. I have changed API keys as well yet it keeps on happening.
This is vannatest app in the cloud
