Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
8ce7052
Replace rcov with simplecov
wijet Oct 21, 2013
1e3c149
Ensure Taps version of OkJson is used to properly encode data.
eric1234 Jul 12, 2013
2c8cca2
Upgrade to latest rack to make sure taps works
wijet Oct 21, 2013
9d1147a
Add sqlite3 as runtime dependency
wijet Oct 21, 2013
154623b
Update sinatra
wijet Oct 21, 2013
4e0734d
switch from Config to RbConfig
michaeljkchoi Mar 31, 2016
7851133
fixed version bug
michaeljkchoi Mar 31, 2016
74d46ed
Support newer Sequel 4.0+
joelvh Mar 29, 2017
c2821c8
Use updated Mocha API
joelvh Mar 29, 2017
a96dd78
Remove unneeded server things
joelvh Mar 29, 2017
0643e6c
Remove patch version in comparison
joelvh Mar 29, 2017
383a08a
Looser version requirements
joelvh Mar 29, 2017
4338a8f
Don't commit `Gemfile.lock` with gem
joelvh Mar 29, 2017
e7f42e0
Remove `build` rake task
joelvh Mar 29, 2017
f681a89
Version bump
joelvh Mar 29, 2017
b1f1fb4
Bugfix: single_integer_primary_key on multi-column PKs https://github…
joelvh Mar 29, 2017
896f76d
Bugfix: always wrote "ETA: 00:00:00" https://github.com/ricardochimal…
joelvh Mar 29, 2017
4db69f9
Fix OkJson::Error rescue (previously referred to undefined constant) …
joelvh Mar 29, 2017
6495471
Fixed the progress bar when pulling the schema https://github.com/ric…
joelvh Mar 29, 2017
f7c352e
Ensure Taps version of OkJson is used to properly encode data https:/…
joelvh Mar 29, 2017
465c2a2
Updated contact info
joelvh Mar 29, 2017
d5b924c
Fixed additional progress bar output
joelvh Mar 29, 2017
258e6ed
Formatting whitespace
joelvh Apr 6, 2017
9f674a6
Cleaned up variables and whitespace
joelvh Apr 7, 2017
d361a07
version bump v0.5.1
joelvh Apr 7, 2017
b283b9d
Start updating README
joelvh Apr 7, 2017
911f537
Updated dependency references and README
joelvh Apr 7, 2017
64b5844
Version bump v0.5.2
joelvh Apr 7, 2017
e79ffad
Be consistent with the session database URI ENV variable, and fallbac…
joelvh Apr 7, 2017
745fa30
Be consistent in using Basic Auth ENV variables when available
joelvh Apr 7, 2017
d82034d
Fix whitespace and unnecessary variables
joelvh Apr 7, 2017
8bd6826
Don't require SQLite3 because you can specify an alternative session DB
joelvh Apr 7, 2017
43ab014
Document ENV variables and options for alternative session DB type
joelvh Apr 7, 2017
40d9114
Version bump v0.5.3
joelvh Apr 7, 2017
a539aa6
Fixed class name in error handling (https://github.com/ricardochimal/…
joelvh Apr 7, 2017
5d8a6cd
Version bump v0.5.4
joelvh Apr 7, 2017
46b0f46
Specify YAML engine (via @espen for issue https://github.com/ricardoc…
joelvh Apr 7, 2017
8a1f40c
Version bump v0.5.5
joelvh Apr 7, 2017
3a73415
Tracking some issues and notes to help further development
joelvh Apr 7, 2017
175a628
Ignore docs
joelvh Apr 7, 2017
e2761da
Fixed links
joelvh Nov 9, 2017
bd90391
Fixed link
joelvh Nov 9, 2017
30d0db3
Rename executable to `taps2`
joelvh Nov 14, 2017
1ae5225
Use Sequel ~> 4.0
joelvh Nov 14, 2017
37a1d6b
Use new executable names
joelvh Nov 14, 2017
3ff2701
Version bump v0.6.0
joelvh Nov 14, 2017
c9c2a7d
Rubocop cleanup
joelvh Nov 15, 2017
6b932a4
Version bump v0.6.1
joelvh Nov 15, 2017
b0586ad
Fix version references
joelvh Nov 15, 2017
0c27d5d
Version bump v0.6.2
joelvh Nov 15, 2017
c72d8ac
Fix version references
joelvh Nov 15, 2017
18fdc36
Make Sequel 5.x compatible
joelvh Nov 15, 2017
1976cc1
Version bump v0.6.3
joelvh Nov 15, 2017
3d363e5
Fix Sequel 5 table creation
joelvh Nov 15, 2017
e9e0279
Version bump v0.6.4
joelvh Nov 15, 2017
fed7111
Fix bin command name for `schema` (now `schema2`)
joelvh Nov 15, 2017
cf7f9ae
Version bump v0.6.5
joelvh Nov 15, 2017
173e512
Use common Sequel version where needed
joelvh Nov 15, 2017
423ea77
Version bump 0.6.6
joelvh Nov 15, 2017
5241f8a
Fix reference to non-existent method in Sequel 5 `Sequel::BasicObject…
joelvh Nov 15, 2017
4e373b8
Version bump v0.6.7
joelvh Nov 15, 2017
9e84240
Fix reference to Rack Multipart since Rack v1.3.x
joelvh Nov 15, 2017
99ed481
Version bump v0.6.8
joelvh Nov 15, 2017
9c6d634
Fix require path
joelvh Nov 15, 2017
3c41797
Version bump v0.6.9
joelvh Nov 15, 2017
8a884f1
Require all of Rack
joelvh Nov 15, 2017
dc9b29a
Version bump v0.6.10
joelvh Nov 15, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,7 @@
*.db
.bundle
pkg
taps-*.gem
taps*.gem
Gemfile.lock
*.dat
rdoc/
15 changes: 4 additions & 11 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,15 +1,8 @@
source "http://rubygems.org"
source 'http://rubygems.org'

# load the gem's dependencies
gemspec

# server-specific dependencies
group :server do
gem "hoptoad_notifier"
gem "extlib", "0.9.15"
gem "mysql", "2.8.1"
gem "mysql2", "0.2.6"
gem "pg", "0.9.0"
gem "thin", "> 1.2.0"
end

# gem 'mysql', '>= 2.8.1'
gem 'mysql2', '>= 0.2.6'
gem 'pg', '>= 0.9.0'
60 changes: 0 additions & 60 deletions Gemfile.lock

This file was deleted.

105 changes: 105 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Taps (2) -- simple database import/export app

A simple database agnostic import/export app to transfer data to/from a remote database.

*Forked and updated* with fixes and improvements. Integrates fixes and updates from [taps-taps](https://github.com/wijet/taps) and [tapsicle](https://github.com/jiffyondemand/tapsicle) forks.

## Installation

Renamed gem

$ gem install taps2

By default, Taps will attempt to create a SQLite3 database for sessions. Unless you specify a different database type, you'll need to install SQLite3. (See _Environment Variables_ for alternative session databases.)

$ gem install sqlite3

Install the gems to support databases you want to work with, such as MySQL or PostgreSQL.

$ gem install mysql2
$ gem install pg

## Configuration: Environment Variables

_All environment variables are optional._

The `TAPS_DATABASE_URL` variable specifies the database URI where session data is stored on CLI or Server.
The `DATABASE_URL` variable is a fallback (and compatible with Heroku) if `TAPS_DATABASE_URL` is not specified.
By default, Taps will create a SQLite3 database.

The `TAPS_LOGIN` and `TAPS_PASSWORD` variables are used for default Basic Authentication.

The `TAPS_YAML_ENGINE` variable lets you specify a YAML engine such as "psych".

The `NO_DEFAULT_DATABASE_URL` variable allows you to require a database URI be sent in the `body` of the request that initiates a Taps session. (Default behavior will use the database URI specified when starting the server.)

The `NO_DUMP_MARSHAL_ERRORS` variable allows you to disable dumping of marshalled data that caused an error.

The `NO_DEFLATE` variable allows you to disable gzip compression (`Rack::Deflater`) on the server.

## Usage: Server

Here's how you start a taps server

$ taps2 server postgres://localdbuser:localdbpass@localhost/dbname httpuser httppassword

You can also specify an encoding in the database url

$ taps2 server mysql://localdbuser:localdbpass@localhost/dbname?encoding=latin1 httpuser httppassword

## Usage: Client

When you want to pull down a database from a taps server

$ taps2 pull postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000

or when you want to push a local database to a taps server

$ taps2 push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000

or when you want to transfer a list of tables

$ taps2 push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000 --tables logs,tags

or when you want to transfer tables that start with a word

$ taps2 push postgres://dbuser:dbpassword@localhost/dbname http://httpuser:httppassword@example.com:5000 --filter '^log_'

## Troubleshooting

* "Error: invalid byte sequence for encoding" can be resolved by adding `encoding` to database URI (https://github.com/ricardochimal/taps/issues/110)
* *Example:* `taps2 server mysql://root@localhost/example_database?encoding=UTF8 httpuser httppassword`
* SQLite3 database URI may require three slashes (e.g. `sqlite3:///path/to/file.db`)
* Make sure to use an absolute/full path to the file on the server

## Known Issues

* Foreign key constraints get lost in the schema transfer
* Indexes may drop the "order" (https://github.com/ricardochimal/taps/issues/111)
* String fields with only numbers may get parsed as a number and lose leading zeros or add decimals (https://github.com/ricardochimal/taps/issues/106)
* Tables without primary keys will be incredibly slow to transfer. This is due to it being inefficient having large offset values in queries.
* Multiple schemas are currently not supported (https://github.com/ricardochimal/taps/issues/97)
* Taps does not drop tables when overwriting database (https://github.com/ricardochimal/taps/issues/94)
* Oracle database classes not fully supported (https://github.com/ricardochimal/taps/issues/89)
* Some blank default values may be converted to NULL in MySQL table schemas (https://github.com/ricardochimal/taps/issues/88)
* Conversion of column data types can cause side effects when going from one database type to another
* MySQL `bigint` converts to PostgreSQL `string` (https://github.com/ricardochimal/taps/issues/77)
* Passwords in database URI can cause issues with special characters (https://github.com/ricardochimal/taps/issues/74)

## Feature Requests

* Allow a single Taps server to serve data from different databases (https://github.com/ricardochimal/taps/issues/103)

## Meta

Maintained by [Joel Van Horn](http://github.com/joelvh)

Written by Ricardo Chimal, Jr. (ricardo at heroku dot com) and Adam Wiggins (adam at heroku dot com)

Early research and inspiration by Blake Mizerany

Released under the MIT License: http://www.opensource.org/licenses/mit-license.php

http://github.com/ricardochimal/taps

Special Thanks to Sequel for making this tool possible http://sequel.rubyforge.org/
51 changes: 0 additions & 51 deletions README.rdoc

This file was deleted.

33 changes: 8 additions & 25 deletions Rakefile
Original file line number Diff line number Diff line change
@@ -1,14 +1,7 @@
task :build do
gemspec = Gem::Specification.load("taps.gemspec")
target = "pkg/#{gemspec.file_name}"

FileUtils.mkdir_p File.dirname(target)
Gem::Builder.new(gemspec).build
FileUtils.mv gemspec.file_name, target
end
require 'simplecov'

begin
require 'rake/rdoctask'
require 'rdoc/task'
Rake::RDocTask.new do |rdoc|
rdoc.rdoc_dir = 'rdoc'
rdoc.title = 'taps'
Expand All @@ -17,33 +10,23 @@ begin
rdoc.rdoc_files.include('lib/**/*.rb')
end
rescue LoadError
puts "Rdoc is not available"
end

begin
require 'rcov/rcovtask'
Rcov::RcovTask.new do |t|
t.libs << 'spec'
t.test_files = FileList['spec/*_spec.rb']
t.verbose = true
end
rescue LoadError
puts "RCov is not available. In order to run rcov, you must: sudo gem install rcov"
puts 'Rdoc is not available'
end

desc "Run all specs; requires the bacon gem"
desc 'Run all specs; requires the bacon gem'
task :spec do
SimpleCov.start if ENV['COVERAGE']
if `which bacon`.empty?
puts "bacon is not available. In order to run the specs, you must: sudo gem install bacon."
puts 'bacon is not available. In order to run the specs, you must: sudo gem install bacon.'
else
system "bacon #{File.dirname(__FILE__)}/spec/*_spec.rb"
end
end

desc "copy/paste env vars for dev testing"
desc 'copy/paste env vars for dev testing'
task :env do
puts "export RUBYLIB='#{File.dirname(__FILE__) + '/lib'}'"
puts "export RUBYOPT='-rrubygems'"
end

task :default => :spec
task default: :spec
File renamed without changes.
5 changes: 0 additions & 5 deletions VERSION.yml

This file was deleted.

26 changes: 18 additions & 8 deletions bin/schema → bin/schema2
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
#!/usr/bin/env ruby

require 'rubygems'
gem 'sequel', '~> 3.20.0'
require_relative '../lib/taps/version'

$:.unshift File.dirname(__FILE__) + '/../lib'
gem 'sequel', Taps::Version.sequel_version

$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'

require 'taps/schema'

cmd = ARGV.shift.strip rescue ''
database_url = ARGV.shift.strip rescue ''
cmd = ARGV.shift.to_s.strip
database_url = ARGV.shift.to_s.strip

def show_usage_and_exit
puts <<EOTXT
Expand All @@ -35,12 +37,20 @@ when 'indexes'
when 'indexes_individual'
puts Taps::Schema.indexes_individual(database_url)
when 'load_indexes'
filename = ARGV.shift.strip rescue ''
indexes = File.read(filename) rescue show_usage_and_exit
filename = ARGV.shift.to_s.strip
indexes = begin
File.read(filename)
rescue StandardError
show_usage_and_exit
end
Taps::Schema.load_indexes(database_url, indexes)
when 'load'
filename = ARGV.shift.strip rescue ''
schema = File.read(filename) rescue show_usage_and_exit
filename = ARGV.shift.to_s.strip
schema = begin
File.read(filename)
rescue StandardError
show_usage_and_exit
end
Taps::Schema.load(database_url, schema)
when 'reset_db_sequences'
Taps::Schema.reset_db_sequences(database_url)
Expand Down
2 changes: 1 addition & 1 deletion bin/schema.cmd → bin/schema2.cmd
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
@ECHO OFF
IF NOT "%~f0" == "~f0" GOTO :WinNT
@"ruby.exe" "./schema" %1 %2 %3 %4 %5 %6 %7 %8 %9
@"ruby.exe" "./schema2" %1 %2 %3 %4 %5 %6 %7 %8 %9
GOTO :EOF
:WinNT
@"ruby.exe" "%~dpn0" %*
2 changes: 1 addition & 1 deletion bin/taps → bin/taps2
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env ruby

$:.unshift File.dirname(__FILE__) + '/../lib'
$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
require 'taps/cli'

Taps::Cli.new(ARGV.dup).run
4 changes: 2 additions & 2 deletions config.ru
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ require 'rubygems'
require 'bundler/setup'
require 'extlib'

$:.unshift File.expand_path("../lib", __FILE__)
$LOAD_PATH.unshift File.expand_path('../lib', __FILE__)
require 'taps/config'

Taps::Config.taps_database_url = ENV['DATABASE_URL']
Taps::Config.taps_database_url = ENV['TAPS_DATABASE_URL'] || ENV['DATABASE_URL']
Taps::Config.login = ENV['TAPS_LOGIN']
Taps::Config.password = ENV['TAPS_PASSWORD']

Expand Down
Loading