You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We need a much smarter way of handling database connections.
Today, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.
For Datasette Library (#417) I want to support potentially hundreds of attached databases. Datasette Edit (#567) is going to introduce a need for writable connections too.
I'd also like to be able to run joins across multiple databases (#283) which further complicates things.
Supporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.
I think what I want it a mechanism where a thread can say "give me a connection for database X" and it either gets back connection to X instantly OR a new connection to X is created and returned OR it blocks (because a certain number of connections to that database exist already) until another thread returns their connection OR it times out and returns an error.
Within this block the current thread has exclusive action to the connection - which has essentially been "checked out" from the pool. When the block ends that connection is made available to be checked out by other threads.
This could accept multiple database names for the case where I want to join across databases:
I'm going to refactor the execute() and execute_against_connection_in_thread() methods.
They currently live on the Datasette class, but in this new world it would make more sense for them to live on the Database, ConnectionGroup or Connection classes.
Activity
simonw commentedon Aug 17, 2019
I think what I want it a mechanism where a thread can say "give me a connection for database X" and it either gets back connection to X instantly OR a new connection to X is created and returned OR it blocks (because a certain number of connections to that database exist already) until another thread returns their connection OR it times out and returns an error.
simonw commentedon Aug 17, 2019
Potential API design:
Within this block the current thread has exclusive action to the connection - which has essentially been "checked out" from the pool. When the block ends that connection is made available to be checked out by other threads.
This could accept multiple database names for the case where I want to join across databases:
First working sketch of connection pool, refs #569
simonw commentedon Oct 2, 2019
I'm going to refactor the
execute()andexecute_against_connection_in_thread()methods.They currently live on the
Datasetteclass, but in this new world it would make more sense for them to live on theDatabase,ConnectionGrouporConnectionclasses.I think I'll put them on the
Databaseclass.simonw commentedon Oct 2, 2019
I'm going to simplify things a bunch by continuing to ignore the cross-database joining issue #283 - I'll post some notes there on my latest thinking.
Refactor .execute() onto Database class
Move .execute() from Datasette to Database
New Pool/ConnectionGroup implementation