Derek furst/disable s3

VERSION

@@ -1 +1 @@
-2.6.11
+2.6.12

docker/entity-api/nginx/conf.d/entity-api.conf

@@ -41,6 +41,17 @@ server {
 
         include uwsgi_params;
         uwsgi_pass uwsgi://localhost:5000;
+
+        # Maximum time nginx will wait to establish a connection to the uwsgi upstream before giving up
+        uwsgi_connect_timeout 10s;
+        # Maximum time nginx will wait between successive writes to uwsgi when sending a request
+        uwsgi_send_timeout    90;
+        # Maximum time nginx will wait for uwsgi to send a response, preventing premature 502/504 errors on legitimately slow requests
+        uwsgi_read_timeout    90;
+        # Size of the buffer used to read the first part of the uwsgi response header
+        uwsgi_buffer_size 32k;
+        # Number and size of buffers used for reading the uwsgi response body, reducing disk buffering warnings for larger responses
+        uwsgi_buffers 4 32k;
     }
 
 }

src/uwsgi.ini

@@ -13,6 +13,9 @@ log-master=true
 master = true
 processes = 12
 
+# Recycles each worker process after handling 2000 requests, preventing gradual memory growth over time
+max-requests = 2000
+
 # Enable the multithreading within uWSGI
 # Launch the application across multiple threads inside each process
 enable-threads = True
@@ -26,3 +29,15 @@ vacuum = true
 
 # Ensure compatibility with init system
 die-on-term = true
+
+# Raises the OS-level socket listen backlog from the default 100, allowing more 
+# connections to queue while workers are busy before new connections are refused
+listen = 512
+
+# Enables the uwsgi stats server on port 9191, exposing a JSON endpoint with 
+# real-time worker and queue metrics for monitoring and diagnostics
+stats = localhost:9191
+
+# Makes the stats server accessible via HTTP rather than a raw socket,
+# allowing standard tools like curl to query it
+stats-http = true

test/locust_test/README.md

@@ -0,0 +1,82 @@
+# Load Testing `entity-api` with Locust
+
+This directory contains our Locust load-testing setup for `entity-api`. While Locust can simulate complex user journeys, our primary goal here is to stress-test our infrastructure to see how different configuration values affect failure points under sustained load. 
+
+Locust provides excellent real-time visualizations for requests over time, response times, and failure rates, making it easy to observe exactly when our services run out of available connections.
+
+## Prerequisites & Setup
+
+1. **Install Dependencies:** Ensure you have the required packages installed via `requirements.txt`.
+   ```bash
+   pip install -r requirements.txt
+   ```
+   *(Note: This installs `locust` and `python-dotenv`)*
+
+2. **Environment Variables:** Create a `.env` file in this directory to store your authorization token.
+   ```env
+   TOKEN=your_actual_bearer_token_here
+   ```
+
+3. **Test Data:** You will need an `ids.json` file in this directory containing an array of valid UUIDs (about 1000 is sufficient). We use real IDs to ensure the API processes the GET requests fully, rather than immediately returning a 404 for a missing ID or a 400 for a malformed one.
+
+## The Locust Task
+
+The current test cases, including the `GetEntityUser` class and the authenticated endpoint tasks, are defined in the local [get_entity_test.py](./get_entity_test.py). 
+
+This script is configured to:
+* Use a `constant(0)` wait time to maximize throughput.
+* Randomly select a UUID from `ids.json` for each request.
+* Inject the required `Authorization` header using your local `.env` file.
+
+## Running the Tests
+
+1. Start the Locust service using the CLI from wherever the locust file is located and give the path to that file:
+   ```bash
+   locust -f path/to/file
+   ```
+2. Open your browser and navigate to the Locust GUI: [http://localhost:8089](http://localhost:8089)
+3. Configure the test parameters in the UI:
+   * **Number of users:** `1000` (Sufficient to force a failure state).
+   * **Spawn rate:** `5` (Users added per second).
+   * **Host:** `http://localhost:xxxx` (The local URL for `entity-api`).
+
+## Observing Results & Identifying Failures
+
+This setup makes it easy to do a control run with no tweaks to the service and see when we hit 502/504 errors. 
+
+When you start the run, switch to the **Charts** tab. Watch the users ramp up. When we've reached the limit of available `uwsgi` connections, there will be a sharp spike in the red line representing failures. You can switch over to the **Failures** tab to confirm that these are `502` errors. Make any desired tweaks to the service configurations, then run again.
+
+## Benchmarks & Configuration Tweaks
+
+*(Note: "Virtual users" here is simply an abstraction; it is just the easiest consistent metric we have to represent the load on the service).*
+
+**Baseline:**
+* Ramping up 5 users/sec with `constant(0)` wait time, the service reliably fails **between 400 and 450 virtual users** on a vm we used for testing; your numbers will likely vary.
+
+**Optimized Configuration:**
+By tweaking `uwsgi` and `nginx` settings, we roughly double the amount of time until failure, reliably reaching **just over 800 virtual users**. 
+
+To replicate these results locally, apply the following tweaks:
+* **uwsgi:** Set `max-requests` to `2000` and increase the `listen` queue value to `512`.
+* **nginx:** Apply the following timeout and buffer parameters to your configuration:
+
+```nginx
+# Maximum time nginx will wait to establish a connection to the uwsgi upstream before giving up
+uwsgi_connect_timeout 10s;
+
+# Maximum time nginx will wait between successive writes to uwsgi when sending a request
+uwsgi_send_timeout    90;
+
+# Maximum time nginx will wait for uwsgi to send a response, preventing premature 502/504 errors on legitimately slow requests
+uwsgi_read_timeout    90;
+
+# Size of the buffer used to read the first part of the uwsgi response header
+uwsgi_buffer_size     32k;
+
+# Number and size of buffers used for reading the uwsgi response body, reducing disk buffering warnings for larger responses
+uwsgi_buffers         4 32k;
+```
+
+## Further Resources
+
+For more information on writing complex tasks, custom load shapes, or advanced reporting, refer to the [Locust Documentation](https://docs.locust.io/en/stable/).

test/locust_test/get_entity_test.py

@@ -0,0 +1,21 @@
+from locust import HttpUser, task, constant
+import json
+import random
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+with open("ids.json") as f:
+    valid_ids = json.load(f)
+
+class GetEntityUser(HttpUser):
+    wait_time = constant(0)
+    token = os.getenv("TOKEN")
+
+    @task
+    def hit_authenticated_endpoint(self):
+        entity_id = random.choice(valid_ids)
+        headers = {"Authorization": f"Bearer {self.token}"}
+        self.client.get(f"/entities/{entity_id}", headers=headers)
+
+

test/locust_test/reindex_entity_test.py

@@ -0,0 +1,21 @@
+from locust import HttpUser, task, constant
+import json
+import random
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+with open("ids.json") as f:
+    valid_ids = json.load(f)
+
+class GetEntityUser(HttpUser):
+    wait_time = constant(0) 
+    token = os.getenv("TOKEN")
+
+    @task
+    def hit_authenticated_endpoint(self):
+        entity_id = random.choice(valid_ids)
+        headers = {"Authorization": f"Bearer {self.token}"}
+        self.client.put(f"/reindex/{entity_id}", headers=headers)
+
+

test/locust_test/requirements.txt

@@ -0,0 +1,2 @@
+locust==2.34.0
+python-dotenv==1.0.1