maple-syrup-group / dbsampler
通过采样操作数据库生成规则化测试数据库
1.0.0
2021-02-01 14:17 UTC
Requires
- doctrine/dbal: ^2.5
- fakerphp/faker: ^1.13
- monolog/monolog: ^1.22||^2.0.0
- pimple/pimple: ^3.0
Requires (Dev)
- friendsofphp/php-cs-fixer: ^2.16
- phpunit/phpunit: ^8.5
- roave/security-advisories: dev-master
- squizlabs/php_codesniffer: ^3.5
README
这是一个通用的工具,用于从数据库中提取和清洗所选表以用作测试数据。在json配置文件的控制下,将一个数据库中的表子集复制到另一个数据库。然后可以将后者数据库导出为SQL以用作测试文件。
用法
- 创建您希望填充的目标数据库。工具只会输出到现有数据库。目标数据库的内容将全部清除。
- 使用DB服务器配置创建
config/credentials.json
。 - 创建
config/*.db.json
文件以定义每个所需数据库的映射 - 运行
bin/dbsampler.php
配置格式
所有配置文件都位于config
子目录中。文件不能包含注释,因为JSON格式不支持此功能,但按照惯例,可能的情况下将忽略名为"comment"的字段。
credentials.json
"driver": "pdo_mysql"
目前被假定,但将来可能会改变。
MySQL
见config/credentials.dist.json
{
"driver": "pdo_mysql",
"dbUser": "root",
"dbPassword": "SOMEPASSWORD",
"dbHost": "127.0.0.1"
}
如果您需要不同的源和目标服务器,这将变为
{
"source": {
"driver": "pdo_mysql",
"dbUser": "root",
"dbPassword": "SOMEPASSWORD",
"dbHost": "sourceDB.example.com"
},
"dest" : {
"driver": "pdo_mysql",
"dbUser": "root",
"dbPassword": "SOMEPASSWORD",
"dbHost": "127.0.0.1"
}
}
如果您需要准备连接,请添加一个initialSql段落
{
"source": {
"driver": "pdo_mysql",
"dbUser": "root",
"dbPassword": "SOMEPASSWORD",
"dbHost": "sourceDB.example.com",
"initialSql": [
"SET NAMES UTF8"
]
},
"dest": {
"driver": "pdo_mysql",
"dbUser": "root",
"dbPassword": "SOMEPASSWORD",
"dbHost": "127.0.0.1",
"initialSql": [
"SET NAMES UTF8",
"SET foreign_key_checks = 0"
]
}
}
Sqlite
见config/credentials.dist.json
{
"driver": "pdo_sqlite",
"directory": "..\/path\/to\/sqlite-dbs"
}
路径假定相对于配置文件,除非它们以'/'开头。要迁移的Sqlite数据库假定是这个目录中的*.sqlite
文件
dbname.db.json
{
"name": "small-sqlite-test", # Configuration name
"sourceDb": "small-source", # Name of the source DB
"destDb": "small-dest", # Name of the destination DB. This DB will get trashed
"tables": { # A set of tables to be copied over. Each table is defined as "table": config
# Every config stanza requires a sampler field. For now, look these up in
# \Quidco\DbSampler\MigrationConfigProcessor::$samplerMap
# All other fields depend on the specific sampler being used; these should
# all be documented in their own class files in src/Sample
"fruits": {
"sampler": "matched",
"constraints": {
"name": [
"apple",
"pear"
]
},
"remember": {
"id": "fruit_ids" # Cross-referencing is supported by "remember" stanzas
# These take the field name of which the values are to be remembered
# matched to a variable name in which the values will be stored
# Note: Variable declarations do not include a '$' symbol
# References MUST be 'remember'ed before being used, there is no
} # dependency resolution here, so order your config appropriately
},
"vegetables": {
"sampler": "NewestById",
"idField": "id",
"quantity": 2
},
"fruit_x_basket": {
"sampler": "matched",
"constraints": {
"fruit_id": "$fruit_ids" # Remembered variables, with $ sign, can be used as cross-references
# This will expand to all ids of the fruits table matched above
},
"where" : [
"basket_id > 1" # The matched sampler can also accept a list of arbitrary WHERE clauses
],
"remember": {
"basket_id": "basket_ids"
}
},
"baskets": {
"sampler": "cleanMatched", # some samplers support field cleaners that are defined in
# \Quidco\DbSampler\FieldCleanerProvider::getCleanerByName
# They modify or replace the content of the field that they are keyed to
"constraints": {
"id": "$basket_ids"
},
"cleanFields": {
"name": "fakefullname"
}
}
},
"views": [ # view support is experimental
"some_view" # views are specified as name only but format may change
] # The destination's CURRENT_USER() is used as the DEFINER for MySQL DBs
}
"Faker" 清洁器
任何不要求参数的 'faker' (fzaninotto/faker) 生成器都可以通过在cleanFields段落中使用"name": "faker:GENERATOR"
直接使用,例如
"cleanFields": {
"ip": "faker:ipv4"
},
扩展项目
此工具设计为通过添加自定义采样器(必须实现\Quidco\DbSampler\SamplerInterface
)和清洁器(在\Quidco\DbSampler\FieldCleanerProvider::getCleanerByName
中记录)进行扩展。
很可能将提供注册外部清洁器和采样器的机制。
目前,仅支持mysql和sqlite数据库,但这也可以扩展。