I wasn’t sure if you had seen this latest study by James King….et al., about the results of D-Wave’s latest 2,000-qubit chip, and their claim that it beat all classical solvers, including yours, by some 2,600x. I know you are very busy and have far important things to worry about, but just thought I should bring it to your attention anyway. Thanks.

https://arxiv.org/abs/1701.04579

Best wishes,

Sol Warda

The relevant bit to this discussion is 27:50-28:20 where Dr Williams says that the new D-Wave chip scales better than one of the algorithms I use (the one I call S3).

I don’t agree with what he says, or at least I think it is misleading, for the reasons given below. (The summary of which is that D-Wave only manages to show a speed improvement as a function of problem size for easy families of problems that take negligible time to solve compared with known hard families.)

To explain a bit first, Williams is quoting, or reproducing, the results of Itay Hen who has devised a family of problem instances for which there is a “planted” optimal solution. This means that even for very large problem instances one can know whether a heuristic solver has found the optimal solution, without the need for an exhaustive (proving) search. It is one of these families that Williams is basing his comparison on. However:

1. The family of problems Williams has chosen is not the most difficult family of problems available using Hen’s technique, because he (Williams) has chosen the clause density to be higher than the hardest value.

2. Even the most difficult problem that can be created using Hen’s technique (using loops, at least – other variations may be different) is relatively easy compared to a simple random +/-1 instance.

3. When the comparison is made between D-Wave and S3 (or S13) using a family of hard Hen instances, D-Wave appears to scale worse, i.e., the time required grows faster, as a function of instance size, for D-Wave than for S3.

These statements can be checked by looking at this paper of Hen et al, where the algorithm I have been calling S3 is called HFS. The Fig. 6 graph shows that for the red lines (high clause density, easy family), D-Wave times are indeed growing more slowly than for HFS, but for the light blue lines (medium clause density, hard family) the D-Wave times are growing faster.

From the abstract, the authors say

“While we do not find evidence for a speedup for the hardest and most frustrated problems in our problem set, we cannot rule out that a speedup might exist for some of the easier, less frustrated problems.” [“Speedup” meaning D-Wave time grows more slowly than classical time.]

It should be said that Williams is using a newer chip (Washington, 1152 qubit) than Hen et al were able to work with, and it is possible that it scales better than the older one because it is less noisy, or due to some other quality improvement, so that is an unknown from my point of view. On the other hand, it is certainly true that S3 (which Williams and Hen have used for testing) is not the best version of my algorithm. S13 (pictured above) scales considerably better, and S14 (treewidth 2) better still. (S13 is better than S3 in absolute terms for all non-trivial sizes, but S14 only becomes better in absolute terms than S13 for about N>=1152.) There are also other (unpublished) classical algorithms that are competitive with, or better than, S13/S14.

]]>Yes, strategy 13 appears to scale (a bit) better than strategy 3. They are similar for N<=392 or so, but S13 is better for larger N.

Did you mean to talk about the times to solution (TTS) being exponentially distributed (which would make the number of solutions found in a time interval Poisson)? You are right to question this, and I think the results above should be adjusted slightly as mentioned below, though I don’t think it will make that much difference.

I found that these times fitted an exponential distribution well, apart from very small problem sizes (N<=72) where it wasn’t such a good fit. (Not so surprising – for larger N, the chance of any one attempt at finding the solution is small, so a solution is the result of many quasi-independent attempts at finding a rare thing.) On the exponential/Poisson assumption, the 99th percentile time should be log(100)*TTS ~= 4.6*TTS, which is what I used for the above graphs (which were made by the file makegraphs2.py in the repository if you want to look).

This 4.6*TTS approximation can be separately tested (as it may hold even when the full Poisson approximation fails) by comparing with the actual 99th percentile time, and on the examples I tried it agrees very well, apart from N<=32 where 4.6*TTS overestimated the 99th percentile time. The reason I didn’t just use the actual 99th percentile time for the graphs was that I had already done the runs before with TTS output only and didn’t want to rerun them, and also because it is more accurate to estimate TTS when there aren’t so many attempts. (I prefer TTS as a measure, but I obviously wanted to make something comparable with the Rønnow paper.)

So summary: I used log(100)*TTS for the above graphs of S13 99th percentile times which I believe are accurate for N>=72, but should be reduced a little for N<=32.

]]>Do you find strategy 13 scales better than plain strategy 3? Also, how are you getting these 99% confidence timings, as I don’t see any options in the code on github (I’ve modified my local copy to time stabletreeexhaust directly). We have found that using precise timing of the stabletreeexhaust function to estimate the actual 99th percentile of the times to solution doesn’t actually match the expectation given from the assumption that the times for a single instance are Poisson distributed. It’s close but not quite right, and varies from instance to instance.

]]>http://www.isi.edu/sites/default/files/top_level/events/aqc2014/day4-talk4-Hen.pdf

]]>