## Background
In the simulator, we do our best to replicate the effects of an
interactive transaction into the simulator's shadow state whenever that
transaction commits.
## Problem
this didn't work:
BEGIN;
ALTER TABLE t ADD COLUMN foo;
DELETE FROM t WHERE bar != 5;
COMMIT;
None of the rows where bar != 5 were deleted because apply_snapshot()
was checking that the rows in the committed table were exactly equal to
the rows that were recorded, but since the recorded deletes contained a
NULL `foo` column, they never matched. This meant that the sim thought
it should still have all the rows that were deleted.
## Fix:
like all the other operations, record add column / drop column too so
that they are applied in sequential order in apply_snapshot()
No explicit test for this - I ran into this in another branch of mine
whose seed doesn't reproduce on main (because I changed the simulator in
that branch).
Reviewed-by: Pedro Muniz (@pedrocarlo)
Closes#4264
this didn't work:
BEGIN;
ALTER TABLE t ADD COLUMN foo;
DELETE FROM t WHERE bar != 5;
COMMIT;
None of the rows where bar != 5 were deleted because apply_snapshot()
was checking that the rows in the committed table were exactly equal
to the rows that were recorded, but since the recorded deletes contained
a NULL `foo` column, they never matched. This meant that the sim thought
it should still have all the rows that were deleted.
Fix:
like all the other operations, record add column / drop column too so
that they are applied in sequential order in apply_snapshot()
No explicit test for this - I ran into this in another branch of mine
whose seed doesn't reproduce on main (because I changed the simulator
in that branch).
After flushing the WAL, the db may attempt an automatic checkpoint. Previously,
a checkpoint failure for any other reason than `Busy` rolled back the transaction;
this is incorrect because the WAL has already been written and synced, so the transaction
is actually committed and durable.
For this reason, do the following decoupling:
- Mark the transaction as committed after `SyncWal`
- Any errors beyond that point are wrapped as `LimboError::CheckpointFailed` which is
handled specially:
* Tx rollback is not attempted in `abort()` - only the WAL locks are cleaned up
and checkpoint state machines are reset
* In the simulator, the results of the query are shadowed into the sim environment
so that the sim correctly assumes that the transaction's effects were in fact
committed.
Bug 1:
apply_snapshot(), which handles committing transaction changes to the
simulator state, had this bug:
let's say a transaction does:
1. ALTER TABLE t ADD COLUMN (e.g., adding 9th column)
2. INSERT INTO t ... (populating all 9 columns)
then apply_snapshot() would:
1. First push the INSERT row (with 9 values) to committed_tables
2. Then detect new_col_count (9) > old_col_count (8) and add NULLs to ALL rows
hence, adding a 10th column to the new row too.
Fix 1:
only add nulls to existing rows that weren't inserted in the same tx after
the ADD COLUMN.
Bug 2:
the exact same thing, but for DROP COLUMN.
Fix 2:
only remove the i'th column from rows that existed prior to the DROP COLUMN
was issued.
When a new table was created within a transaction, we were incorrectly clearing the rows of the new table when committing.
This resulted in false positives where the sim thought there shouldn't be any
rows in the table.
Fix: don't do that.
e.g. in "ReadYourUpdatesBack", we first do an UPDATE and then a
SELECT to verify that those updates were actually made.
Problem is the SELECT assertion was made even if the UPDATE failed.
Fix: change interaction handling so that if a precondition in a
property fails, the entire property is abandoned.
the issue was that the sim was executing queries on
commit, instead of applying their effects. for example,
if transaction T1 did a `DELETE FROM t WHERE TRUE` it would
delete all the rows even if another mvcc transaction T2 had
inserted rows concurrently that should not be visible to
T1.
WriteWriteConflict should rollback in memory tx but not fail sim.
TxError should not rollback tx but not fail sim either.
Neither of the above should cause sim to shadow the query.
prompt to opus 4.5:
Create a parser that generates a Rust integration test a lá test_transactions.rs
from files like .bugbase/17016242926153013649/plan.sql
the "-- <integer>" means connection number, so as many connections should be created
as there are distinct suffixes like that at the end of a line. lines BEGINNING with --
eg "-- begin testing 'ReadYourUpdatesBack'" can be ignored.
the end result should be that the parser tool prints out a valid rust file containing
the necessary imports and a test function containing the statements (modeled after
test_transactions.rs tests).
this should be a lightweight script or CLI tool in the simulator directory where you
can give it a sql path and it will produce that test.
This PR is a working doc on a roadmap for the simulator. @pedrocarlo
@LeMikaelF please take a look.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3954
to calculate metrics per generation step
- simplify generation as we now only store `Interaction`. So now we can
funnel most of the logic for interaction generation, metric update,
and Interaction append in the `PlanGenerator::next`.
Modify `generation/property.rs` to use the Builder
- add additional metadata to `Interaction` to give more context for
shrinking and iterating over interactions that originated from the
same interaction.
- add Iterator like utilities for `InteractionPlan` to facilitate
iterating over interactions that came from the same property:
will track `Interaction` instead of `Interactions` in the Plan, this
change will impossibilitate the serialization of the InteractionPlan with Serde Json.
- make --load just load the previous cli args
