Skip to content

Compare Marconi Utxo with cardano-db-sync - #44

Draft
kayvank wants to merge 1 commit into
input-output-hk:mainfrom
kayvank:5383/compare-marconi-utxo-with-db-sync
Draft

Compare Marconi Utxo with cardano-db-sync#44
kayvank wants to merge 1 commit into
input-output-hk:mainfrom
kayvank:5383/compare-marconi-utxo-with-db-sync

Conversation

@kayvank

@kayvank kayvank commented May 23, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Compare Marconi Utxo with those of cardano-db-sync. The approach is :

  • stream utxo-view from cardano-db-sync to sqlite
  • compute the marconi-utxos stored in sqlite
  • verify that marconi-utxos store in sqlite are a subset of those in cardano-db-sync

[PLT-5383]

Pre-submit checklist:

  • Branch
    • Tests are provided (if possible)
    • Commit sequence broadly makes sense and have useful messages
    • Important changes are reflected in changelog.d of the affected packages
    • Relevant tickets are mentioned in commit messages
  • PR
    • (For external contributions) Corresponding issue exists and is linked in the description
    • Targeting main unless this is a cherry-pick backport
    • Self-reviewed the diff
    • Useful pull request description
    • If relevant, reference the ADR in the PR and reference the PR in the ADR
    • Reviewer requested

@kayvank
kayvank force-pushed the 5383/compare-marconi-utxo-with-db-sync branch from aa5f2e9 to 5e42a9b Compare May 23, 2023 04:37
@kayvank
kayvank force-pushed the 5383/compare-marconi-utxo-with-db-sync branch 10 times, most recently from 4491214 to 6082c8f Compare June 4, 2023 04:37
@kayvank
kayvank marked this pull request as ready for review June 4, 2023 04:45
@kayvank
kayvank requested review from berewt, eyeinsky and koslambrou June 4, 2023 04:57
@berewt

berewt commented Jun 5, 2023

Copy link
Copy Markdown
Contributor

Are the comparison result available somewhere?

berewt
berewt approved these changes Jun 5, 2023
View reviewed changes

@berewt berewt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Did you put the comparison somewhere?

Comment thread marconi-chain-index/test-compare-cardano-db-sync/Spec.hs Outdated
3. Run this test by setting the env variables:

- CARDANO_NODE_SOCKET_PATH
- CARDANO_NODE_SOCKET_PATH:

@berewt berewt Jun 5, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fat fingers?

@kayvank
kayvank force-pushed the 5383/compare-marconi-utxo-with-db-sync branch 2 times, most recently from f6e4947 to 512fa13 Compare June 6, 2023 15:11
@kayvank

kayvank commented Jun 6, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

the test create a deltaAddresses table that we expect to have no elements.
The table proces that all addresses in marconi up to a slotNo are in dbsync

So, if you run the test, which takes ~600 seconds, for fully synched preprod, the SQLite tables you may check are:

  • deltaAddresses
  • marconiUtxos
  • dbSyncUtxos

I can do run of this locally to demo it?

@kayvank
kayvank force-pushed the 5383/compare-marconi-utxo-with-db-sync branch from 512fa13 to 4ac7241 Compare June 6, 2023 15:52
@kayvank

kayvank commented Jun 6, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Are the comparison result available somewhere?

Here are the results from running the tests:

  Utxo
    Compare all Utxo addresses between Marconi and cardano-db-sync: Address Utxo Marconi and cardano-db-sync comparason where maximum slotNo = 29786631
number Utxos in marconi database: 750397
number Utxos in cardano-db-sync database: 751279
delta addresses between marconi and cardano-db-sync: 0
OK (312.01s)
        ✓ marconiUtxoAddressesAreSubsetOfDbSyncUtxosAddresses passed 1 test.

All 1 tests passed (312.01s)

@kayvank
kayvank force-pushed the 5383/compare-marconi-utxo-with-db-sync branch 2 times, most recently from 0925a4e to 204b6c5 Compare June 6, 2023 16:18
@kayvank
kayvank marked this pull request as draft June 7, 2023 16:06
Kayvan Kazeminejad
Compare Marconi Utxo with cardano-db-sync
b458b18 
Stream db-sync utxos to marconi SQLite and compute delta of addresses,
addresses in Marconi but not in cardano-db-sync

[PLT-5383]

compare dbsync
@kayvank
kayvank force-pushed the 5383/compare-marconi-utxo-with-db-sync branch from 204b6c5 to b458b18 Compare June 23, 2023 03:27
@kayvank
kayvank marked this pull request as ready for review June 28, 2023 13:12
@kayvank
kayvank requested a review from berewt June 28, 2023 13:15
berewt
berewt reviewed Jun 29, 2023
View reviewed changes
Comment thread marconi-chain-index/test-compare-cardano-db-sync/Spec.hs
- CARDANO_NODE_CONFIG_PATH
- DBSYNC_PG_URL
- MARCONI_DB_DIRECTORY_PATH
- DBSYNC_PGPASSWORD: The default password for cardano-db-sync's postgres database is in its repo in the file: config/secrets/postgres_password

@berewt berewt Jun 29, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why isn't it required anymore?

@kayvank kayvank Jul 14, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the long delay.
I am using URL, connection string for PG stuff, It is One environment variable rather than userid and password.

berewt
berewt approved these changes Jun 29, 2023
View reviewed changes

@berewt berewt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good to me have you stored the result of the comparaison somewhere?

kayvank
kayvank commented Jul 16, 2023
View reviewed changes
Comment thread marconi-chain-index/test-compare-cardano-db-sync/Utxo.hs
}
deriving (Eq, Show)

-- | print the list of offending addresses. These are addresses in db-sync and not in marconi that have utxos

@kayvank kayvank Jul 16, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@berewt the discrepancies are stored in the delta table in sqlite.
If there are any discrepancies, they are also printed

@koslambrou
koslambrou marked this pull request as draft September 5, 2023 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@berewt berewt berewt approved these changes
@eyeinsky eyeinsky Awaiting requested review from eyeinsky
@koslambrou koslambrou Awaiting requested review from koslambrou

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kayvank @berewt