Skip to content

More advanced connection pooling #569

Open

Description

@simonw

We need a much smarter way of handling database connections.

Today, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.

For Datasette Library (#417) I want to support potentially hundreds of attached databases. Datasette Edit (#567) is going to introduce a need for writable connections too.

I'd also like to be able to run joins across multiple databases (#283) which further complicates things.

Supporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.

Activity

  1. simonw commented on Aug 17, 2019

    @simonw
    OwnerAuthor

    I think what I want it a mechanism where a thread can say "give me a connection for database X" and it either gets back connection to X instantly OR a new connection to X is created and returned OR it blocks (because a certain number of connections to that database exist already) until another thread returns their connection OR it times out and returns an error.

  2. simonw commented on Aug 17, 2019

    @simonw
    OwnerAuthor

    Potential API design:

    with pool.connection("fixtures") as conn:
        conn.set_time_limit(1000)
        conn.allow_all()
        conn.execute(...)

    Within this block the current thread has exclusive action to the connection - which has essentially been "checked out" from the pool. When the block ends that connection is made available to be checked out by other threads.

    This could accept multiple database names for the case where I want to join across databases:

    with pool.connection("fixtures", "dogs") as conn:
        conn.execute(...)
  3. simonw commented on Oct 2, 2019

    @simonw
    OwnerAuthor

    I'm going to refactor the execute() and execute_against_connection_in_thread() methods.

    They currently live on the Datasette class, but in this new world it would make more sense for them to live on the Database, ConnectionGroup or Connection classes.

    I think I'll put them on the Database class.

  4. simonw commented on Oct 2, 2019

    @simonw
    OwnerAuthor

    I'm going to simplify things a bunch by continuing to ignore the cross-database joining issue #283 - I'll post some notes there on my latest thinking.

  5. added a commit that references this issue on Oct 2, 2019
  6. added this to the Datasette 1.0 stable milestone on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions