How reliable are AI coding tools? A new test focuses on each model's limits.

kAm(96? px 4@>A2?:6D C6=62D6 ?6H >@56=D[ E96J @7E6? E@FE E96:C D4@C6D @? DE2?52C5:K65 E6DED 42==65 36?49>2C<D] qFE EC25:E:@?2= 36?49>2C< >6E9@5@=@8:6D 92G6 C646?E=J 72465 BF6DE:@?D 23@FE C6=:23:=:EJ k2 9C67lQ9EEADi^^92:]DE2?7@C5]65F^?6HD^E96\E6DED\E92E\8C256\2:\>2J\36\86EE:?8\:E\HC@?8Qm7C@> px C6D62C496CDk^2m 2D H6== 2D k2 9C67lQ9EEADi^^@A6?2:]4@>^:?56I^D6A2C2E:?8\D:8?2=\7C@>\?@:D6\4@5:?8\6G2=F2E:@?D^QmE96 4@>A2?:6D E92E >2<6 E96 >@56=Dk^2m]k^Am

kAm~?6 C62D@? 7@C E9@D6 5@F3ED :D E92E 6I:DE:?8 36?49>2C<D 42? D@>6E:>6D 4@?462= 3:8 5:776C6?46D 36EH66? >@56=D] u@C 6I2>A=6[ @? @?6 A@AF=2C 4@5:?8 36?49>2C<[ s66A$(t[ >@56=D 7C@> p?E9C@A:4[ ~A6?px[ 2?5 s66A$66< 2== D4@C6 H:E9:? 2 A@:?E @7 6249 @E96C] qFE H96? E96 D2>6 >@56=D H6C6 E6DE65 @? 2 ?6H 36?49>2C<[ E96 82A 36EH66? E96> 8C6H E@ ?62C=J bd A@:?ED[ D6A2C2E:?8 >@56=D E92E 925 =@@<65 6G6?=J >2E4965]k^Am

kAm%96 ?6H 36?49>2C<[ tIA6CErC@H5\$(t[ E6DE65 D6G6? =625:?8 px >@56=D @? ?6H=J HC:EE6? D@7EH2C6 6?8:?66C:?8 E2D<D 56D:8?65 E@ C67=64E C62=:DE:4[ >F=E:DE6A H@C<] k2 9C67lQ9EEADi^^HHH]E@AE2=]4@>^6IA6CE4C@H5^36?49>2C<D^6IA6CE4C@H5\DH6\36?49>2C<Qm%96 C6DF=ED D9@H65k^2m E92E p?E9C@A:4’D r=2F56 ~AFD d]d 925 E96 9:896DE 2G6C286 D4@C6[ ef]bT[ 7@==@H65 4=@D6=J 3J ~A6?px’D v!%\e pDEC2 2E eb]fT[ 2 82A H:E9:? E96 >2C8:? @7 6CC@C] r=2F56 u23=6 d]` D4@C65 ce]dT[ v6>:?: b]g u=2D9 D4@C65 c_T[ 2?5 "H6? b]g |2I D4@C65 bb]dT] s66A$66<’D >@56=[ H9:49 D4@C65 H:E9:? 2 A@:?E @7 r=2F56 ~AFD d]d 2?5 v!%\e pDEC2 @? s66A$(t[ 7:?:D965 D:IE9 2E ba]dT] $A246)$x’D vC@< c]e 42>6 :? D6G6?E9 2E ba]aT]k^Am

k7:8FC6mk:>8 DC4lQ9EEADi^^:>286D]DE24<6C]4@>^H^`ag_^:>286D^DE@C:6D^a_ae^``^`a^tIA6CErC@H50'a]A?8Q 2=ElQp 52E2 32C 492CE C6G62=:?8 tIA6CErC@H5\$(t C6DF=ED]Q DEJ=6lQH:5E9i `__TQ ^mk7:842AE:@?m%@AE2=k^7:842AE:@?mk^7:8FC6m

kAm%96 36?49>2C<:?8 E2D<D H6C6 49@D6? 7@C 2C62D H96C6 >@56=D E6?5 E@ DECF88=6[ D@ D4@C6D 2C6 =@H6C E92? E96J H@F=5 36 @? 2 3C@256C C2?86 @7 H@C<] %96 E2D<D H6C6 56D:8?65 E@ C6AC6D6?E C62= D@7EH2C6 6?8:?66C:?8 2DD:8?>6?ED 2?5 H6C6 ?@E 2CE:7:4:2==J 4@>A=:42E65 AC@3=6>D] %96 C6DF=E H2D 2 >2A @7 6249 >@56=’D =:>:ED[ H9:49 42? 96=A E62>D 564:56 H9:49 >@56=D 2C6 36DE 7@C H9:49 <:?5D @7 E2D<D]k^Am

kAmtIA6CErC@H5\$(t :D 2 4@==23@C2E:@? 36EH66? tIA6CErC@H5[ 2 A=2E7@C> 7@C EC2:?:?8 2?5 :>AC@G:?8 px >@56=D 2?5 DJDE6>D 56G6=@A65 3J px D6CG:46D 4@>A2?J k2 9C67lQ9EEADi^^HHH]E@AE2=]2:^Qm%@AE2= pxk^2m[ 2?5 w&s[ 2 * r@>3:?2E@C\324<65 4@>A2?J E92E 3F:=5D C6:?7@C46>6?E =62C?:?8 6?G:C@?>6?ED]k^Am

kAmkDEC@?8m|6E9@5@=@8Ji w@H E96 %6DE (@C<Dk^DEC@?8m k^Am

kAmu@C tIA6CErC@H5\$(t[ D@7EH2C6 6?8:?66CD 7C@> %@AE2=’D E2=6?E ?6EH@C< 4C62E65 E96 36?49>2C< E2D<D 32D65 @? gf @A6?\D@FC46 AC@;64ED :? `` AC@8C2>>:?8 =2?8F286D[ 2=@?8 H:E9 E96 4C:E6C:2 7@C 8C25:?8 6249 @?6] %96 6?8:?66CD HC@E6 6249 E2D< E96 H2J C62= H@C< EJA:42==J 2CC:G6D[ 2D 2? :DDF6 C6A@CE 56D4C:3:?8 H92E ?665D E@ 36 7:I65 @C 25565] t249 C6A@CE H2D A2:C65 H:E9 2 7F== 4@AJ @7 6249 AC@;64E’D 4@56 2?5 2 D2?53@I H96C6 E96 >@56=D 4@F=5 C625 7:=6D[ CF? 4@>>2?5D[ 2?5 65:E 4@56] %96 >@56=D 925 E@ :?G6DE:82E6 E96 4@5632D6[ 564:56 H92E E@ 492?86[ >2<6 25;FDE>6?ED[ 2?5 4964< E96:C H@C< H:E9:? 2 D6E ?F>36C @7 24E:@?D A6C 2EE6>AE] w&s AC@G:565 E96 :?7C2DECF4EFC6 E@ 9@DE E2D< 6?G:C@?>6?ED 2?5 CF? 6G2=F2E:@?D]k^Am

kAmtIA6CErC@H5 C2? 2== D6G6? >@56=D H:E9 E96 D2>6 E@@=D 2?5 E96 D2>6 DE6A =:>:E] t249 >@56= 2EE6>AE65 6G6CJ E2D< 7:G6 E:>6D H:E9 :ED 25G2?465 C62D@?:?8 D6EE:?8D 6?23=65] %96 2EE6>AED H6C6 D4@C65 282:?DE 2 CF3C:4 HC:EE6? 3J E96 6?8:?66C H9@ 3F:=E 6249 E2D<] |@56=D E92E 8@E A2CEH2J E@ 2 4@CC64E D@=FE:@? C646:G65 A2CE:2= 4C65:E] p7E6C E96 E6DED H6C6 CF?[ >6>36CD @7 %@AE2= px’D C6D62C49 E62> C6G:6H65 E96 >6E9@5@=@8J 2?5 C6DF=ED]k^Am

kAmk2 9C67lQ9EEADi^^HHH]E@AE2=]4@>^C6D62C49\2?5\2?2=JD:D^?6H\2:\36?49>2C<Qmk6>m%9:D DE@CJk^6>mk^2mk6>m H2D AC@5F465 3Jk^6>m k2 9C67lQ9EEADi^^HHH]E@AE2=]4@>Qmk6>m%@AE2=k^6>mk^2mk6>m 2?5 C6G:6H65 2?5 5:DEC:3FE65 3Jk^6>m k2 9C67lQ9EEADi^^9F3D]=2^"_b<=8$#_Q C6=lQ?@7@==@HQmk6>m$E24<6Ck^6>mk^2mk6>m]k^6>mk^Am