I tested Composer 2.5 Fast vs GPT-5.5 Thinking on a real CVE that came out 3 days ago-> Drupal CVE-2026-9082, a critical SQL injection. Each run was a fresh session, new folder and file names, so that model wont recognize. Both models found the bug all 3 runs. Composer 2.5 Fast:- Run 1: nailed it. Gave a working exploit payload. Rated it High severity. Run 2: missed the point. Called it a "Low severity input validation issue. A real Audit would deprioritize this. Run 3: nailed it again. Even caught a bonus issue GPT missed. GPT-5.5 Thinking:- All 3 runs: detected the bug, rated it "Medium", never gave an exploit payload. Run 3 said "less likely full SQL injection" -> basically the same dismissal Composer had in Run 2. Note: Actual CVE rating is "Highly Critical". Neither model got that right. Composer 2.5, runs gave detailed report but its variance is the real story benchmarks don't show. One run nails the exploit. The next run tells you to ignore it. 2nd Test: Tested Composer 2.5 Fast vs GPT-5.5 Thinking vs Gemini 3.5 Flash on a real Linux kernel CVE that came out 3 weeks ago -> CVE-2026-31700, a TOCTOU race condition in net/packet/af_packet.c. Each run was a fresh session, new folder and file names, so that model wont recognize. 3 runs per model, 9 runs total. T The actual bug: vnet_hdr points into mmap'd memory shared with userspace. Kernel validates it once, then re-reads it later. Userspace can modify between validation and use. Race conditions are notoriously hard for LLMs because there is no pattern in the code, the bug lives in execution timing. Composer 2.5 Fast:- Run 1: nailed it. Severity Critical (slightly overrated, actual is High). 10 issues total. Run 2: nailed it again. CWE-367, exact lines, 3-step exploit flow. Even cross-referenced CVE-2025-38617. Run 3: best run of all 9. Explicitly named CVE-2026-31700, stated the actual upstream fix, found 2 bonus TOCTOU variants. GPT-5.5 Thinking:- Run 1: clean pass. Identified TOCTOU on mmap shared ring. Severity High (exactly matches CVSS 7.8). Run 2: complete failure. Found OOB read, MTU bypass, integer underflow. Never said "race" or "TOCTOU". Run 3: recovered. Solid pass. Gemini 3.5 Flash:- 0 for 3. Three different wrong CVEs across three runs. Historical AF_PACKET integer overflow, then signed/unsigned conversion, then CVE-2017-7308. Confident, detailed, wrong every time. @skcd42 @yunta_tsai @mntruell @amanrsanger @sualehasif996
Post summary
The author tested LLMs on two recent CVEs, showing that some runs produced exploit payloads and detailed technical info, but no evidence of active exploitation or patching was provided.