Sage: Ticket #23913: Doctest failures with GMP
https://trac.sagemath.org/ticket/23913
<p>
With GMP is used instead of MPIR, there are two doctest failures because of error checking that GMP does but MPIR does not:
</p>
<pre class="wiki">**********************************************************************
File "src/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper
Failed example:
1 << (2^60)
Expected:
Traceback (most recent call last):
...
MemoryError: failed to allocate ... bytes
Got:
gmp: overflow in mpz type
Traceback (most recent call last):
File "/home/jdemeyer/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/jdemeyer/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module>
Integer(1) << (Integer(2)**Integer(60))
File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (build/cythonized/sage/rings/integer.c:39270)
return (<Integer>x)._shift_helper(y, 1)
File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (build/cythonized/sage/rings/integer.c:39014)
sig_on()
RuntimeError: Aborted
**********************************************************************
</pre><p>
and
</p>
<pre class="wiki">sage -t --warn-long 61.3 src/sage/ext/memory.pyx
**********************************************************************
File "src/sage/ext/memory.pyx", line 9, in sage.ext.memory
Failed example:
2^(2^63-2)
Expected:
Traceback (most recent call last):
...
MemoryError: failed to allocate 1152921504606847008 bytes
Got:
gmp: overflow in mpz type
Traceback (most recent call last):
File "/home/jdemeyer/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/jdemeyer/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.ext.memory[0]>", line 1, in <module>
Integer(2)**(Integer(2)**Integer(63)-Integer(2))
File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (build/cythonized/sage/rings/integer.c:14173)
sig_on()
RuntimeError: Aborted
**********************************************************************
</pre><p>
Upstream GMP provides no way to hook this error: <a class="ext-link" href="https://gmplib.org/list-archives/gmp-discuss/2017-September/006144.html"><span class="icon"></span>https://gmplib.org/list-archives/gmp-discuss/2017-September/006144.html</a>
</p>
en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/23913
Trac 1.1.6fbisseyThu, 21 Sep 2017 23:16:58 GMTcc set
https://trac.sagemath.org/ticket/23913#comment:1
https://trac.sagemath.org/ticket/23913#comment:1
<ul>
<li><strong>cc</strong>
<em>fbissey</em> added
</li>
</ul>
TicketmderickxMon, 25 Sep 2017 20:07:02 GMTcc changed
https://trac.sagemath.org/ticket/23913#comment:2
https://trac.sagemath.org/ticket/23913#comment:2
<ul>
<li><strong>cc</strong>
<em>mderickx</em> added
</li>
</ul>
TicketmderickxMon, 25 Sep 2017 20:23:58 GMT
https://trac.sagemath.org/ticket/23913#comment:3
https://trac.sagemath.org/ticket/23913#comment:3
<p>
Since both mpir and gmp are optional packages couldn't we also implement a fix by having
</p>
<pre class="wiki">2^(2^63-2) #optional mpir
some error
2^(2^63-2) #optional gmp
another error
</pre><p>
in the mean time? If someone here at this ticket thinks that it is a good idea, then I will create a ticket with the relatively easy patch for it.
</p>
TicketfbisseyMon, 25 Sep 2017 20:34:28 GMT
https://trac.sagemath.org/ticket/23913#comment:4
https://trac.sagemath.org/ticket/23913#comment:4
<p>
That's an interesting idea considering that by default tests are run with <code>--optional=mpir,python2,sage</code> so the default behavior isn't changed. But I am not sure what happens when <code>gmp</code> is selected. But I think that is a good approach.
</p>
TicketmderickxMon, 25 Sep 2017 21:06:31 GMT
https://trac.sagemath.org/ticket/23913#comment:5
https://trac.sagemath.org/ticket/23913#comment:5
<p>
The default argument for optional is actually determined programmatically. <code>sage</code> is always there and the rest is just a list of optional packages. It just happens that python2 and mpir are optional packages that are installed by default. If you would build sage with python3 and gmp instead it will be <code>--optional=gmp,python3,sage</code> instead.
</p>
TicketfbisseyMon, 25 Sep 2017 21:08:10 GMT
https://trac.sagemath.org/ticket/23913#comment:6
https://trac.sagemath.org/ticket/23913#comment:6
<p>
I was not sure of the mechanics but it makes sense for it be that way. Your solution should work. I'll be waiting to see your suggested branch.
</p>
TicketmderickxTue, 26 Sep 2017 21:29:25 GMTcommit, branch, author set
https://trac.sagemath.org/ticket/23913#comment:7
https://trac.sagemath.org/ticket/23913#comment:7
<ul>
<li><strong>commit</strong>
set to <em>beef1a813eecf7d24de8f05ec839f824dc5275ee</em>
</li>
<li><strong>branch</strong>
set to <em>public/23913</em>
</li>
<li><strong>author</strong>
set to <em>Maarten Derickx</em>
</li>
</ul>
<p>
Ok, I created a branch with how I think the fix should look like. But I did not put it to needs review yet because I do not understand how to check that it works, the main problem is that I do not now how to properly build sage with gmp.
</p>
<p>
I tried just:
</p>
<pre class="wiki">sage -i gmp
</pre><p>
in a sage that was already build since I was hoping to save time, but this does not work (it is running the optional gmp test and not the mpir one, but the test result is that of mpir). Should I do:
</p>
<pre class="wiki">sage -i gmp
make build
</pre><p>
Or do I need to do:
</p>
<pre class="wiki">make distclean
make gmp
make build
</pre><hr />
<p>
New commits:
</p>
<table class="wiki">
<tr><td><a class="ext-link" href="https://git.sagemath.org/sage.git/commit?id=beef1a813eecf7d24de8f05ec839f824dc5275ee"><span class="icon"></span>beef1a8</a></td><td><code>Trac #23913: Fixed doctest failures with GMP</code>
</td></tr></table>
TicketfbisseyTue, 26 Sep 2017 21:34:41 GMT
https://trac.sagemath.org/ticket/23913#comment:8
https://trac.sagemath.org/ticket/23913#comment:8
<p>
I am not sure from an existing sage I am afraid, rebuilding is necessary and there could be stray libraries in the way. It would better and much cleaner - but unfortunately more time consuming - to do
</p>
<pre class="wiki">./configure --with-mp=gmp
make
</pre>
TicketmderickxTue, 26 Sep 2017 21:36:21 GMTstatus changed
https://trac.sagemath.org/ticket/23913#comment:9
https://trac.sagemath.org/ticket/23913#comment:9
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_info</em>
</li>
</ul>
TicketmderickxTue, 26 Sep 2017 21:44:45 GMTstatus changed
https://trac.sagemath.org/ticket/23913#comment:10
https://trac.sagemath.org/ticket/23913#comment:10
<ul>
<li><strong>status</strong>
changed from <em>needs_info</em> to <em>needs_work</em>
</li>
</ul>
<p>
I got an error that configure was not found in a clean install, so I had to do
</p>
<pre class="wiki">make configure
./configure --with-mp=gmp
make
</pre><p>
I am now building and waiting for how the tests turn out. If you already have an install with gmp lying around and you test this branch there then you will probably know the results before I do.
</p>
TicketmderickxTue, 26 Sep 2017 21:54:40 GMTattachment set
https://trac.sagemath.org/ticket/23913
https://trac.sagemath.org/ticket/23913
<ul>
<li><strong>attachment</strong>
set to <em>mpfr-3.1.5.p0.log</em>
</li>
</ul>
TicketmderickxTue, 26 Sep 2017 22:00:08 GMT
https://trac.sagemath.org/ticket/23913#comment:11
https://trac.sagemath.org/ticket/23913#comment:11
<p>
The commands:
</p>
<pre class="wiki">make configure
./configure --with-mp=gmp
make
</pre><p>
resulted in the failure building of mpfr on OS X 10.12.4 with Xcode 8.3.3 before gcc was build. I've attached the log, so I still don't know how to test it on this machine. I can test it on Ubuntu tomorrow.
</p>
TicketfbisseyTue, 26 Sep 2017 22:01:23 GMT
https://trac.sagemath.org/ticket/23913#comment:12
https://trac.sagemath.org/ticket/23913#comment:12
<p>
That's still a problem thanks for pointing it out. We didn't notice it on the <code>gmp</code> upgrade ticket but we should have.
</p>
TicketfbisseyTue, 26 Sep 2017 22:09:48 GMT
https://trac.sagemath.org/ticket/23913#comment:13
https://trac.sagemath.org/ticket/23913#comment:13
<p>
Ok, I have now noticed something. You are not using the latest <code>gmp</code> from the <code>gmp</code> upgrade ticket. It would be best if that was the case. Since it is not in a beta yet you should depend on <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a> and merge its branch with the one in this ticket. It looks like there is an issue with the produced libgmp with your version of xcode and it is quite possible it is fixed by the gmp version of <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a>.
</p>
TicketmderickxTue, 26 Sep 2017 23:15:56 GMT
https://trac.sagemath.org/ticket/23913#comment:14
https://trac.sagemath.org/ticket/23913#comment:14
<p>
It took me longer then it should to figure out how to pull <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a>, but it now works and gets me past mpfr. The build is now at the point of building gcc. I'll report back tomorrow.
</p>
TicketfbisseyTue, 26 Sep 2017 23:17:14 GMT
https://trac.sagemath.org/ticket/23913#comment:15
https://trac.sagemath.org/ticket/23913#comment:15
<p>
Good, that means that <code>gmp</code> before the upgrade was broken on OS X anyway, so it was high time to upgrade it.
</p>
TicketjdemeyerWed, 27 Sep 2017 09:11:20 GMT
https://trac.sagemath.org/ticket/23913#comment:16
https://trac.sagemath.org/ticket/23913#comment:16
<p>
Your patch makes sense, but you will need to special-case 32-bit. I'm pretty sure that the 32-bit result is an <code>OverflowError</code> both with <code>mpir</code> and with <code>gmp</code>.
</p>
TicketmderickxWed, 27 Sep 2017 17:30:57 GMT
https://trac.sagemath.org/ticket/23913#comment:17
https://trac.sagemath.org/ticket/23913#comment:17
<p>
Thanks for pointing this out, I am now building sage with the gmp from <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a> on a 32bit machine as well to see what the exact error message looks like.
</p>
TicketjdemeyerWed, 27 Sep 2017 17:33:44 GMT
https://trac.sagemath.org/ticket/23913#comment:18
https://trac.sagemath.org/ticket/23913#comment:18
<p>
The <code>OverflowError</code> is raised by Cython, so it would be the same for <code>mpir</code> and <code>gmp</code>.
</p>
TicketgitThu, 28 Sep 2017 00:38:17 GMTcommit changed
https://trac.sagemath.org/ticket/23913#comment:19
https://trac.sagemath.org/ticket/23913#comment:19
<ul>
<li><strong>commit</strong>
changed from <em>beef1a813eecf7d24de8f05ec839f824dc5275ee</em> to <em>dcb984a4f99eefbfbd5064bf82fda815482c601d</em>
</li>
</ul>
<p>
Branch pushed to git repo; I updated commit sha1. New commits:
</p>
<table class="wiki">
<tr><td><a class="ext-link" href="https://git.sagemath.org/sage.git/commit/?id=dcb984a4f99eefbfbd5064bf82fda815482c601d"><span class="icon"></span>dcb984a</a></td><td><code>Trac 23913: Fixed doctest failures with GMP in 32-bit</code>
</td></tr></table>
TicketmderickxThu, 28 Sep 2017 01:02:24 GMTstatus changed
https://trac.sagemath.org/ticket/23913#comment:20
https://trac.sagemath.org/ticket/23913#comment:20
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
</ul>
<p>
Ok, I tested the current branch on:
</p>
<ul><li>32-bit gmp + <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a>
</li><li>32-bit mpir
</li><li>64-bit mpir
</li></ul><p>
I still need to test 64-bit gmp + <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a> since the build on my laptop got interrupted a few times, I think it should pass but I am not 100% sure. I am hesitant to put it into needs review before I know this case passes, however if either of you can verify that this case also passes then feel free to review this ticket.
</p>
<p>
I don't think this ticket needs a dependency on <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a> since it merges cleanly with it.
</p>
TicketmderickxThu, 28 Sep 2017 04:29:31 GMTcommit, branch changed
https://trac.sagemath.org/ticket/23913#comment:21
https://trac.sagemath.org/ticket/23913#comment:21
<ul>
<li><strong>commit</strong>
changed from <em>dcb984a4f99eefbfbd5064bf82fda815482c601d</em> to <em>bf7056499f7e4c9a7fca342523238a27144cc8c9</em>
</li>
<li><strong>branch</strong>
changed from <em>public/23913</em> to <em>public/23913_clean</em>
</li>
</ul>
<p>
Ok the case: 64-bit gmp + <a class="closed ticket" href="https://trac.sagemath.org/ticket/19706" title="enhancement: Upgrade GMP (closed: fixed)">#19706</a> failed. I didn't realise that the doctesting framework only sees errors as errors if the error is thrown on the first line.
</p>
<p>
So the expected/got in
</p>
<pre class="wiki">Expected:
Traceback (most recent call last):
...
MemoryError: failed to allocate ... bytes
Got:
gmp: overflow in mpz type
Traceback (most recent call last):
File "/home/jdemeyer/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/jdemeyer/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module>
Integer(1) << (Integer(2)**Integer(60))
File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (build/cythonized/sage/rings/integer.c:39270)
return (<Integer>x)._shift_helper(y, 1)
File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (build/cythonized/sage/rings/integer.c:39014)
sig_on()
RuntimeError: Aborted
</pre><p>
Is a bit misleading since for the passing doctest for this is without the "gmp: overflow in mpz type" line.
</p>
<pre class="wiki"> Traceback (most recent call last):
...
RuntimeError: Aborted
</pre><p>
anyway. It now passes in all 4 cases for me, so ready for review.
</p>
<hr />
<p>
New commits:
</p>
<table class="wiki">
<tr><td><a class="ext-link" href="https://git.sagemath.org/sage.git/commit?id=bf7056499f7e4c9a7fca342523238a27144cc8c9"><span class="icon"></span>bf70564</a></td><td><code>Trac #23913: Fixed doctest failures with GMP</code>
</td></tr></table>
TicketjdemeyerThu, 28 Sep 2017 20:34:59 GMTstatus changed; reviewer set
https://trac.sagemath.org/ticket/23913#comment:22
https://trac.sagemath.org/ticket/23913#comment:22
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
set to <em>Jeroen Demeyer</em>
</li>
</ul>
TicketvbraunSun, 01 Oct 2017 00:18:58 GMTstatus, branch changed; resolution set
https://trac.sagemath.org/ticket/23913#comment:23
https://trac.sagemath.org/ticket/23913#comment:23
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
<li><strong>branch</strong>
changed from <em>public/23913_clean</em> to <em>bf7056499f7e4c9a7fca342523238a27144cc8c9</em>
</li>
</ul>
Ticket