test: end-to-end roundtrip tests comparing decomposition vs direct Ondsel #31

New Issue

forbes · 2026-02-20T22:28:23Z

forbes commented

2026-02-20 22:28:23 +00:00

Summary

Build a comprehensive test suite that validates the decomposition solver against the Ondsel backend by solving identical assemblies with both and comparing results.

Context

The decomposition solver is a meta-solver that wraps Ondsel — its results must be equivalent (within tolerance) to solving the same problem monolithically. Roundtrip tests are the primary correctness validation: if decompose → dispatch → reconcile produces the same placements as direct Ondsel, the decomposition is correct.

These tests require either the real kcsolve module (FreeCAD build) or carefully constructed mock solvers that replicate Ondsel's behavior for known test cases.

Depends on: #30 (DecompositionSolver — the complete pipeline to test)

Test assemblies

Tier 1: Minimal cases (2-3 parts)

Two parts + Fixed joint — trivially well-constrained, single subproblem
Two parts + Revolute — 1 DOF remaining, single subproblem
Two parts + Ball joint — 3 DOF remaining, single subproblem
Three parts in chain (A—rev—B—rev—C) — decomposes into 2 subproblems at articulation point B

Tier 2: Medium assemblies (5-10 parts)

Linear chain of 5 parts — 4 biconnected components, sequential solve
Star topology (1 hub + 4 spokes) — hub is articulation point, 4 independent subproblems
Loop (4-bar linkage: A—B—C—D—A) — triconnected, no decomposition benefit
Mixed: chain + independent cluster — tests Stage A (connected components) + Stage B (biconnected)

Tier 3: Stress cases (50-100 parts)

Deep chain of 50 parts — validates O(k·m³) vs O(n³) scaling advantage
Wide star with 100 spokes — validates parallel-independent subproblem handling
Overconstrained assembly — Fixed + Revolute between same parts → diagnostics match

Tier 4: Diagnostic validation

Redundant constraint detection — 3 parallel constraints → same redundancy identified by both
Under-constrained assembly — Ball joint only → same DOF count from both
Mixed well/over/under — complex assembly with regions of each type

Test structure

class TestDecompositionVsOndsel:
    """Compare decomposition solver against direct Ondsel for identical inputs."""

    def _solve_both(self, ctx: SolveContext) -> tuple[SolveResult, SolveResult]:
        """Solve with both solvers, return (decomp_result, ondsel_result)."""
        decomp = DecompositionSolver(backend_id="ondsel")
        ondsel = kcsolve.load("ondsel")
        return decomp.solve(ctx), ondsel.solve(ctx)

    def _assert_placements_equal(self, a: SolveResult, b: SolveResult, tol=1e-6):
        """Assert all part placements match within tolerance."""
        ...

    def _assert_diagnostics_compatible(self, a: SolveResult, b: SolveResult):
        """Assert diagnostic findings are consistent (decomp may find more)."""
        ...

Benchmark tests

In addition to correctness, measure timing:

Record solve() wall time for both solvers on each test assembly
For chain/star assemblies of increasing size (10, 25, 50, 100 parts), plot scaling
The decomposition solver should show sub-linear scaling for decomposable topologies

Benchmarks should be marked with @pytest.mark.benchmark and not run in CI by default.

Tasks

Build SolveContext fixtures for all 14 test assemblies
Implement _solve_both() helper (with mock fallback when kcsolve unavailable)
Implement placement comparison with configurable tolerance
Implement diagnostic compatibility check
Tier 1 tests (4 cases)
Tier 2 tests (4 cases)
Tier 3 tests (3 cases)
Tier 4 diagnostic tests (3 cases)
Benchmark tests with @pytest.mark.benchmark
CI integration: run Tier 1-2 in CI, Tier 3-4 marked as slow/optional

Acceptance criteria

All Tier 1-2 tests pass: decomposition results match Ondsel within 1e-6 tolerance
Tier 3 stress tests pass with relaxed tolerance (1e-4) for large assemblies
Tier 4 diagnostic tests: decomposition finds at least the same issues as Ondsel
Benchmark shows decomposition solver scaling advantage for chain/star topologies >20 parts

## Summary Build a comprehensive test suite that validates the decomposition solver against the Ondsel backend by solving identical assemblies with both and comparing results. ## Context The decomposition solver is a meta-solver that wraps Ondsel — its results must be equivalent (within tolerance) to solving the same problem monolithically. Roundtrip tests are the primary correctness validation: if decompose → dispatch → reconcile produces the same placements as direct Ondsel, the decomposition is correct. These tests require either the real `kcsolve` module (FreeCAD build) or carefully constructed mock solvers that replicate Ondsel's behavior for known test cases. Depends on: #30 (DecompositionSolver — the complete pipeline to test) ## Test assemblies ### Tier 1: Minimal cases (2-3 parts) 1. **Two parts + Fixed joint** — trivially well-constrained, single subproblem 2. **Two parts + Revolute** — 1 DOF remaining, single subproblem 3. **Two parts + Ball joint** — 3 DOF remaining, single subproblem 4. **Three parts in chain** (A—rev—B—rev—C) — decomposes into 2 subproblems at articulation point B ### Tier 2: Medium assemblies (5-10 parts) 5. **Linear chain of 5 parts** — 4 biconnected components, sequential solve 6. **Star topology** (1 hub + 4 spokes) — hub is articulation point, 4 independent subproblems 7. **Loop** (4-bar linkage: A—B—C—D—A) — triconnected, no decomposition benefit 8. **Mixed: chain + independent cluster** — tests Stage A (connected components) + Stage B (biconnected) ### Tier 3: Stress cases (50-100 parts) 9. **Deep chain of 50 parts** — validates O(k·m³) vs O(n³) scaling advantage 10. **Wide star with 100 spokes** — validates parallel-independent subproblem handling 11. **Overconstrained assembly** — Fixed + Revolute between same parts → diagnostics match ### Tier 4: Diagnostic validation 12. **Redundant constraint detection** — 3 parallel constraints → same redundancy identified by both 13. **Under-constrained assembly** — Ball joint only → same DOF count from both 14. **Mixed well/over/under** — complex assembly with regions of each type ## Test structure ```python class TestDecompositionVsOndsel: """Compare decomposition solver against direct Ondsel for identical inputs.""" def _solve_both(self, ctx: SolveContext) -> tuple[SolveResult, SolveResult]: """Solve with both solvers, return (decomp_result, ondsel_result).""" decomp = DecompositionSolver(backend_id="ondsel") ondsel = kcsolve.load("ondsel") return decomp.solve(ctx), ondsel.solve(ctx) def _assert_placements_equal(self, a: SolveResult, b: SolveResult, tol=1e-6): """Assert all part placements match within tolerance.""" ... def _assert_diagnostics_compatible(self, a: SolveResult, b: SolveResult): """Assert diagnostic findings are consistent (decomp may find more).""" ... ``` ## Benchmark tests In addition to correctness, measure timing: - Record `solve()` wall time for both solvers on each test assembly - For chain/star assemblies of increasing size (10, 25, 50, 100 parts), plot scaling - The decomposition solver should show sub-linear scaling for decomposable topologies Benchmarks should be marked with `@pytest.mark.benchmark` and not run in CI by default. ## Tasks - [ ] Build SolveContext fixtures for all 14 test assemblies - [ ] Implement `_solve_both()` helper (with mock fallback when kcsolve unavailable) - [ ] Implement placement comparison with configurable tolerance - [ ] Implement diagnostic compatibility check - [ ] Tier 1 tests (4 cases) - [ ] Tier 2 tests (4 cases) - [ ] Tier 3 tests (3 cases) - [ ] Tier 4 diagnostic tests (3 cases) - [ ] Benchmark tests with `@pytest.mark.benchmark` - [ ] CI integration: run Tier 1-2 in CI, Tier 3-4 marked as slow/optional ## Acceptance criteria - All Tier 1-2 tests pass: decomposition results match Ondsel within 1e-6 tolerance - Tier 3 stress tests pass with relaxed tolerance (1e-4) for large assemblies - Tier 4 diagnostic tests: decomposition finds at least the same issues as Ondsel - Benchmark shows decomposition solver scaling advantage for chain/star topologies >20 parts

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kindred/solver#31