Migrating to new DB due to crashes

Hi all,

I've been on sqlite3 since maybe foreman 1.1, but now updated to 1.5.1 and
with almost 50 nodes it can't keep up (SQLite3::SQLException: cannot start
a transaction within a transaction: begin transaction)

So I started the migration process based on the documentation but I ran
into many different postgresql issues (ident authentication failure,
foreman role didn't exist, dropping some unused tables from sqlite etc) At
the end the prod2dev script was still failing and I was concerned about the
cleanliness of the environment so I reverted back to my snapshot.

I was wondering if anyone had the same issues or have a clear step by step
method to migrate the sqlite3 db to postgresql or mysql. I don't care about
the reports and statistics but the ENC data would be important to maintain.

Thanks a lot,
Peter

Looks like it works now, in case anyone needs it or want to expand the doc,
here is the full process:

Install (and start) the required packages

yum install postgresql-server
service postgresql initdb
service postgresql start
chkconfig postgresql on
yum install foreman-postgresql

Set up the initial foreman DB and user:

sudo -u postgres psql template1
template1=# alter user postgres with encrypted password 'password';
template1=# create database foreman;
template1=# create user foreman with encrypted password 'password';
template1=# grant all privileges on database foreman to foreman;

Drop the old puppet tables from sqlite:

sqlite3 /var/lib/foreman/db/production.sqlite3
> drop table resources;
> drop table puppet_tags;
> drop table resource_tags;
> drop table source_files;
> drop table param_names;
> drop table param_values;
> drop table inventory_nodes;
> drop table inventory_facts;

Configure foreman database access:

vi /etc/foreman/database.yml
development:
adapter: postgresql
database: foreman
username: foreman
password: password
host: localhost

test:
adapter: sqlite3
database: db/test.sqlite3
pool: 5
timeout: 5000

production:
adapter: sqlite3
database: db/production.sqlite3
pool: 5
timeout: 5000

Configure postgresql authentication with encrypted passwords:

vi /var/lib/pgsql/data/pg_hba.conf

"local" is for Unix domain socket connections only

local all all md5

IPv4 local connections:

host all all 127.0.0.1/32 md5

IPv6 local connections:

host all all ::1/128 md5

Now run the commands as per the documentation:

sudo -u postgres psql -c 'ALTER USER foreman WITH SUPERUSER'
foreman-rake db:migrate RAILS_ENV=production
foreman-rake db:migrate RAILS_ENV=development
foreman-rake db:convert:prod2dev
sudo -u postgres psql -c 'ALTER USER foreman WITH NOSUPERUSER'

cat <<EOF > reset.sql
SELECT 'SELECT SETVAL(' ||quote_literal(S.relname)|| ', MAX('
>>quote_ident(C.attname)|| ') ) FROM ' ||quote_ident(T.relname)|| ';'
FROM pg_class AS S, pg_depend AS D, pg_class AS T, pg_attribute AS C
WHERE S.relkind = 'S'
AND S.oid = D.objid
AND D.refobjid = T.oid
AND D.refobjid = C.attrelid
AND D.refobjsubid = C.attnum
ORDER BY S.relname;
EOF

psql -U postgres -Atq -f reset.sql -o temp foreman
psql -U postgres -f temp foreman
rm temp reset.sql

Finally, swap the 'production' and 'development'

vi /etc/foreman/database.yml
production:
adapter: postgresql
database: foreman
username: foreman
password: password
host: localhost

test:
adapter: sqlite3
database: db/test.sqlite3
pool: 5
timeout: 5000

development:
adapter: sqlite3
database: db/production.sqlite3
pool: 5
timeout: 5000

Hello,

glad to read that.

> Looks like it works now, in case anyone needs it or want to expand the doc,
> here is the full process:

You can contribute to our documentation directly via github.

https://github.com/theforeman/theforeman.org/tree/gh-pages/_includes/manuals/1.5

Section Upgrade might get new chapter perhaps.

> SELECT 'SELECT SETVAL(' ||quote_literal(S.relname)|| ', MAX('
> >>quote_ident(C.attname)|| ') ) FROM ' ||quote_ident(T.relname)|| ';'
> FROM pg_class AS S, pg_depend AS D, pg_class AS T, pg_attribute AS C
> WHERE S.relkind = 'S'
> AND S.oid = D.objid
> AND D.refobjid = T.oid
> AND D.refobjid = C.attrelid
> AND D.refobjsubid = C.attnum
> ORDER BY S.relname;
> EOF

What is this by the way? ^^^

··· -- Later,

Lukas “lzap” Zapletal

Hi Lukas,

That section is from the official foreman documentation, special notes for
postgresql. Apparently prod2dev messes up the psql sequence numbers and
this code would fix it. There is a more detailed page here:
http://wiki.postgresql.org/wiki/Fixing_Sequences

Cheers,
Peter

··· On Thursday, 10 July 2014 03:09:10 UTC-7, Lukas Zapletal wrote: > > Hello, > > glad to read that. > > > Looks like it works now, in case anyone needs it or want to expand the > doc, > > here is the full process: > > You can contribute to our documentation directly via github. > > > https://github.com/theforeman/theforeman.org/tree/gh-pages/_includes/manuals/1.5 > > > Section Upgrade might get new chapter perhaps. > > > SELECT 'SELECT SETVAL(' ||quote_literal(S.relname)|| ', MAX(' > > >>quote_ident(C.attname)|| ') ) FROM ' ||quote_ident(T.relname)|| ';' > > FROM pg_class AS S, pg_depend AS D, pg_class AS T, pg_attribute AS C > > WHERE S.relkind = 'S' > > AND S.oid = D.objid > > AND D.refobjid = T.oid > > AND D.refobjid = C.attrelid > > AND D.refobjsubid = C.attnum > > ORDER BY S.relname; > > EOF > > What is this by the way? ^^^ > > -- > Later, > > Lukas "lzap" Zapletal >