Skip to content

feat: Add MySQL and MariaDB support for SqlStorageClient#1749

Open
Mantisus wants to merge 5 commits intoapify:masterfrom
Mantisus:add-mysql
Open

feat: Add MySQL and MariaDB support for SqlStorageClient#1749
Mantisus wants to merge 5 commits intoapify:masterfrom
Mantisus:add-mysql

Conversation

@Mantisus
Copy link
Collaborator

Description

  • Add MySQL and MariaDB support for SqlStorageClient.

Issues

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds MySQL and MariaDB support to SqlStorageClient, expanding the database backend options beyond SQLite and PostgreSQL. The implementation includes database-specific configurations, SQL dialect handling, and special error handling for MariaDB's unique locking behavior.

Changes:

  • Added sql_mysql optional dependency group with aiomysql, cryptography, and SQLAlchemy
  • Implemented MySQL/MariaDB-specific SQL operations (INSERT IGNORE, ON DUPLICATE KEY UPDATE)
  • Added MariaDB-specific error handling for locked rows (error 1020)
  • Updated documentation to include installation and configuration instructions for MySQL/MariaDB

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
uv.lock Added aiomysql 0.3.2, cryptography 46.0.5, and pymysql 1.1.2 dependencies; updated package metadata to include sql-mysql extra
pyproject.toml Added sql_mysql optional dependency group with required packages
src/crawlee/storage_clients/_sql/_storage_client.py Extended dialect validation to include mysql/mariadb; added MySQL/MariaDB-specific engine configuration (connect_timeout, READ COMMITTED isolation level)
src/crawlee/storage_clients/_sql/_request_queue_client.py Reorganized transaction handling; enabled skip_locked for MySQL/MariaDB in fetch operations
src/crawlee/storage_clients/_sql/_client_mixin.py Implemented MySQL/MariaDB insert-with-ignore and upsert statements; added MariaDB error 1020 handling for locked rows
docs/guides/storage_clients.mdx Updated documentation to include MySQL/MariaDB in supported databases; added installation instructions and configuration warnings

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Mantisus and others added 2 commits February 13, 2026 18:21
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant