SWE-bench Multimodal scores do not depend on images, at least in my ablations with GPT-6 Luna

I have been establishing baselines for a project at the prime intellect residency and we were using swe-multimodal v2 for performing ablations. I found out that gpt 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was bit baffling at first but based on some further slop-investigation, I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, swe-multimodal is not really testing the multimodality. In this post, I will briefly go through the methodology of the evaluation and the investigation itself. MethodologyThe benchmark has 480 tasks built from GitHub issues in JavaScript repos, most with screenshots. The tasks are relatively focused, UI oriented and possible to do within 20 minutes: in the no-image run, Luna's median time per task was about 5.5 minutes. I was using GPT 6 luna xhigh for ablations. Cheapest model and higher…