Add Python 3.12 / 3.13 / 3.14 support#122
Open
Martin-Molinero wants to merge 14 commits into
Open
Conversation
(cherry picked from commit caac33d)
(cherry picked from commit e10d333)
(cherry picked from commit e976558)
(cherry picked from commit 65af098)
Python 3.14 introduced a new assertion that prevents us from using PyObject_GenericSetAttr directly in our meta type. To work around this, we manipulate the type dict directly. This workaround is a simplified variant of Cython's workaround from cython/cython#6325. The relevant Python change is in python/cpython#118454 (cherry picked from commit 08550d0)
(cherry picked from commit f3face0)
(cherry picked from commit 8dfe408)
Not at all sure why this helps, but when assigning `None` instead, the object is gone at the time of garbage collection. (cherry picked from commit 8e0333d)
In Python 3.14, the objects __dict__ seems to already be half deconstructed, leading to crashes during garbage collection. Since gc in Python is single-threaded (I think :)), it should be fine to have a single static for this. If that is not true, we can always use a thread-local instead. (cherry picked from commit 908e13b)
* Use non-BOM encodings The documentation of the used `PyUnicode_DecodeUTF16` states that not passing `*byteorder` or passing a 0 results in the first two bytes, if they are the BOM (U+FEFF, zero-width no-break space), to be interpreted and skipped, which is incorrect when we convert a known "non BOM" string, which all strings from C# are. (cherry picked from commit 195cde6)
Python 3.12 eagerly normalizes the error indicator, so PyErr_Fetch now hands us the SyntaxError instance (whose str() omits the offending source line) instead of the raw args tuple (whose str() included it). Callers that surface PythonException.Message for compile diagnostics therefore lost the offending source text on 3.12+. GetMessage now re-appends the SyntaxError 'text' attribute when present. This is a no-op on <=3.11 (there the fetched value is a tuple without the SyntaxError attributes) and only affects SyntaxError messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three CPython behavior changes surfaced as failures/crashes once the
overload-resolution crash was fixed, all on 3.12+:
- ClassManagerTests.BindsCorrectOverloadForClassName crashed the host with
"Python memory allocator called without holding the GIL". TestClass2's
Get(PyObject o) re-enters Python via ToPython() while MethodBinder has
released the GIL (allow_threads) around the managed call. A managed
callback that re-enters Python must re-acquire the GIL; tolerated on
<=3.11, fatal on 3.12+. Wrap the body in using (Py.GIL()).
- TestGetsPythonCodeInfoInStackTrace[ForNestedInterop]: 3.12+ adds caret
indicator lines (e.g. "~~~^^^") under source lines in tracebacks, shifting
the positional assertions. Drop caret-only lines before asserting
(no-op on <=3.11).
- Codecs.ExceptionDecodedNoInstance: 3.12 eagerly normalizes exceptions, so
the error indicator always carries an instance ("value"); the instanceless
scenario this decoder targets can no longer be produced. Guard the test to
<3.12.
Verified: full embed suite green on 3.11 (910/910) with these changes; the
three previously-failing tests pass on 3.14.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fork resolves PyTypeObject field offsets from the hardcoded
TypeOffset{major}{minor} tables (it does not run geninterop at build),
so a missing table makes ABI.Initialize throw "Python ABI v... is not
supported" and every test on that version fails at PythonEngine init.
Only 3.6-3.11 and 3.14 tables were present.
Vendor the 3.12 and 3.13 tables from pythonnet/pythonnet upstream
(byte-identical to upstream master; same source as the already-present
TypeOffset314) and add 3.12 + 3.13 to the CI matrix.
Local verification (uv standalone CPython 3.13, this branch's fixes):
embed suite ABI-initializes correctly and runs 847 passed / 0 failed
(parity with 3.14). 3.12 table is vendored from the same authoritative
source; CI exercises it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Martin-Molinero
changed the title
…load The embed-test suite re-initializes the interpreter per fixture. On CPython 3.13/3.14 the suite aborted with "Failed to import encodings module" - not a filesystem problem (strace shows the file opens fine) but interpreter import state corrupted across re-initialization. Root cause isolated to a single test: TestPythonEngineProperties.SetPythonPath. It uses PythonEngine.PythonPath, which pins a fixed module search path via the deprecated Py_SetPath. CPython 3.13+ keeps that path config in _PyRuntime across Py_Finalize and offers no way to reset it back to auto-computation without the PyConfig API, so once this test runs every later re-initialization in the same process is forced onto the pinned path and eventually cannot bootstrap encodings. All other fixtures - including the normal Initialize/Shutdown cycles in pyinitialize and TestFinalizer - run fine in a single process. Run only SetPythonPath in its own test process so it cannot pollute the rest of the suite. Verified locally: full embed suite green on 3.11, 3.13 and 3.14 (main run 908 / SetPythonPath 1, 0 failures); no regression. Also delete TestDomainReload: AppDomain reload is not supported on modern .NET (single-domain), so those tests (MarshalByRefObject / AppDomain.CreateDomain) are obsolete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Martin-Molinero
force-pushed
the
cherry/python-314-support
branch
from
35b9d14 to
9718399
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the fork up to date with modern CPython (3.12, 3.13, 3.14) on top of the existing 3.8–3.11 support. CI is green on all of 3.8 … 3.14.
Changes
PyThreadState_GetUnchecked(replacing the removed_PyThreadState_UncheckedGet), removal ofPyUnicode_AsUnicodeusage, non-BOM UTF encodings.tp_clear: moved the type-clear reentry guard to the .NET side (the previous Python-dict guard could be collected during clear, crashing init on 3.14).TypeOffset312/TypeOffset313/TypeOffset314tables from upstream (the fork resolvesPyTypeObjectfield offsets from hardcodedTypeOffset{maj}{min}tables; missing tables makeABI.Initializethrow "ABI not supported").ToPython()) whileMethodBinderhas released the GIL is now GIL-safe; tolerated on ≤3.11, fatal on 3.12+.3.8 … 3.14. The embed-test re-initialization-heavy fixtures (TestPythonEngineProperties/pyinitialize/TestFinalizer) run in a separate test process — CPython 3.13+ does not fully support repeatedPy_Finalize/Py_Initialize, and accumulating ~40 per-fixture re-inits in one process eventually corrupts interpreter state ("Failed to import encodings module"; strace confirms the file opens fine — it is interpreter state, not the filesystem). Isolating those fixtures keeps each process under the threshold.TestDomainReloadis dropped: AppDomain reload is not supported on modern .NET.Status
✅ Green on 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14.