I wrote in my last post that I want to move this blog from WordPress to MODx and so I thought this would be a good real world lesson in using xPDO. Rather than custom coding a database migration tool using plain old php like I've done in the past, I thought it would be cool to see what xPDO could do, how easy it would be, and how robust xPDO is. I've spent 2 hours on the code below and have never used xPDO before, and so far I'm pretty impressed with what xPDO can do. It's nice to work with a lightweight ORM tool that just does it's thing without extra framework code to wade through. I've used the Doctrine ORM in Symfony for quite a while and its a bit complex a bloated (do we really need DQL?), so compared to that, I found xPDO faster and easier to learn.
First, I want to explain my goals for moving this blog to MODx Revolution. I've used MODx for the past three years and it's time for me to pony up and go all the way with MODx. I've written a lot on this blog that MODx is great, but it isn't great blog software and that's why I use WordPress. However, after seeing what Revolution can do, I really believe it changes the game and could actually catch up to WordPress sooner than we think. Also, using WordPress doesn't help MODx become a better blogging tool, and that's what I'd really like to do.
I've also had some trouble getting WordPress to do what I want it to do on this site. This site is more than just a blog - it has a tools section with custom PHP code and a tutorials section. I feel I've had to hack WordPress to get these things done. The entire time I was coding the templates and custom plugin code to do this, it was always in the back of my mind how easy this would be to do in MODx. So I've been stuck between having lots of great blog functionality with WordPress, and wanting all the the customization that is so easy with MODx. I think I can have both with Revolution.
Below is the beginning of this move, and I'm sharing code at this early state to get feedback from the MODx community and just to share my experience with anyone who is interested. I'm interested to find out if what I'm doing is a good way to go about it, so if you have suggestions, feel free to comment. The goals of the migration are:
I'm assuming that you have a local LAMP, XAMPP, or MAMP stack set up and running. WAMP most likely will not work and gave me headaches for a long time with Revolution. Just use XAMPP on Windows and you'll be fine I think it's a better, more up-to-date product anyway.
Besides the dev environment, you need two things to get started: 1) an export of your wordpress data using phpMyAdmin's export feature, or one from mysqldump, 2) a running installation of MODx Revolution (I'm currently using MODx Revolution 2.0.0-rc-1 rev6614).
First you'll need to create a database for your wordpress data. Mine is called 'wordpress_devtrench' and you'll see it referenced in the code below. Once you have a database created, simply import your wordpress database into it. You don't need to install WordPress since we just need the data.
Next, install a fresh copy of Revolution. The database I set up is called 'modx_devtrench', and you'll see that referenced below as well. All database logins are on my local machine so using 'root' with no password fine, just don't do that on your live server.
After getting MODx set up and my WP database installed, the first thing I did was to create a schema and model based on my existing WP database. Thanks to Jason Coward, this was very easy:
<?php
/*
* xPDO Wordpress to MODx Migration Code
* Author James Ehly (devtrench.com)
*
* This code is part of the WP to MODx migration and should go in your
* core/model/schema directory. Simply run it from a web browser or cli and
* it will create the wordpress schema from the database and all of the
* xpdo objects in core/model/wordpress
*
* Once this is done we can run the wp2modx.php script that does the actual
* data migration.
*
* @todo some manual intervention is needed to set up the table relationships we need
*
* Thanks to Jason Coward for posting how do set up a schema from a database:
* http://modxcms.com/forums/index.php/topic,16562.0.html
*/
$mtime= microtime();
$mtime= explode(" ", $mtime);
$mtime= $mtime[1] + $mtime[0];
$tstart= $mtime;
//Customize this line based on the location of your script
include_once (dirname(dirname(dirname(__FILE__))) . '/xpdo/xpdo.class.php');
// set your database credentials to use your wordpress database, and set the wordpress prefix (default is 'wp_')
$xpdo= new xPDO('mysql:host=localhost;dbname=wordpress_devtrench','root','','wp_');
// Set the package name and root path of that package
$xpdo->setPackage('modx', XPDO_CORE_PATH . '../model/');
$xpdo->setDebug(true);
$manager= $xpdo->getManager();
$generator= $manager->getGenerator();
//Use this to create a schema from an existing database - this will write the wordpress.mysql.schema.xml file
$xml= $generator->writeSchema(XPDO_CORE_PATH . '../model/schema/wordpress.mysql.schema.xml', 'wordpress', 'xPDOObject', 'wp_');
//Use this to generate classes and maps from your schema
// NOTE: by default, only maps are overwritten; delete class files if you want to regenerate classes
// this will generate all of the model code in core/model/wordpress/
$generator->parseSchema(XPDO_CORE_PATH . '../model/schema/wordpress.mysql.schema.xml', XPDO_CORE_PATH . '../model/');
$mtime= microtime();
$mtime= explode(" ", $mtime);
$mtime= $mtime[1] + $mtime[0];
$tend= $mtime;
$totalTime= ($tend - $tstart);
$totalTime= sprintf("%2.4f s", $totalTime);
echo "\nExecution time: {$totalTime}\n";
Basically this code creates a schema file (an xml file that describes the database structure), and a folder called 'wordpress' in core/model/. Inside of the wordpress folder will be all of the model class files that we need to migrate our data using xpdo.
Now that we have an xpdo model to work with that we can start to create the code that will migrate WP data into MODx. Since the databases are separate, we need 2 xPDO objects: the MODx object and an xPDO object with the WordPress package loaded.
Once we have the objects created it's simply a matter of getting all the posts and making a loop where we do the mapping between objects. xPDO makes this so easy with it's getters and setters. The code is commented pretty well so just go line by line and read it.
<?php
/*
* xPDO Wordpress to MODx Migration Code
* Author James Ehly (devtrench.com)
*
* This script uses xpdo to connect to modx and wordpress packages to migrate
* data from a wordpress database to a modx database
*
* The code below may or may not work for your particular wordpress and MODx install.
* It should be used as a reference point, and you should customize it to your
* specific needs, but I hope that most general cases are covered.
*
* This script is intended to be run on a new install of MODx Revolution. If
* you are using an existing MODx Revo install then the script will probably
* work but might have some unexpected results. So, backup your data first, and
* use at your own risk.
*
*/
// Include the xpdo and modx classes
include ('core/xpdo/xpdo.class.php');
include ('core/model/modx/modx.class.php');
// Instantiate a new modx object. MODx inherits from xpdo so we can use it
// like an xpdo object, but it has the extra functions needed for saving content.
// Thanks to Shaun McCormick for writing docs on this.
$modx = new modX();
$modx->initialize('web');
// Now instantiate a new xpdo object and add our wordpress package. This gives
// us the ability to make queries on the wordpress database as an xpdo object.
$wp = new xPDO('mysql:host=localhost;dbname=wordpress_devtrench;charset=utf-8', 'root', '');
$wp->addPackage('wordpress','core/model/','wp_');
// get all wordpress posts. Isn't this so easy?
$posts = $wp->getCollection('Posts');
// iterate over each post and create a new modResource object, mapping our post
// fields to our wordpress fields
foreach($posts as $post)
{
$resource = '';
$resource = $modx->newObject('modResource',array(
'content'=>$post->get('post_content'),
'pagetitle'=>$post->get('post_title'),
'context_key'=>'web',
'alias'=>$post->get('post_name'),
'published'=> ($post->get('post_status') == 'publish') ? 1 : 0,
'pub_date'=> ($post->get('post_status') == 'publish') ? $post->get('post_date') : 0,
)
);
// call the save function which inserts our object record into the database.
$resource->save();
}
// we're done
echo 'done.';
?>
This code is pretty simple so far but it works. After running this script, all of my WordPress posts, pages, and versions are set up as document resources in MODx and I can see and edit them in the manager. Not bad for 2 hours of research/learning and coding. The next step will be to create the relationships in my WordPress schema (this has to be done by hand). Once that is done I'll be able to start to map items related to posts to Template Variables related to my documents. I also plan to set up Quip to handle comments on the MODx side and figure out how to migrate all of my WP comments. The appropriate logic will also need to written to handle how WP posts, pages and versions should be migrated to MODx. For instance, I want my blog posts to be under a main blog page, not under the site root, and I want pages to be directly under the site root (pretty much reverse of what it is now). Versions will probably just be dropped.
First impressions are always important, and xPDO has really impressed me so far. It's turning out to be exactly what I've wanted in MODx for so long; a simple way to get MODx to interact with custom data. I'm excited to see where this project takes me with MODx Revolution and xPDO.
Comments (12)
May 20, 2010 at 06:36 PM
Jay Gilmore:
James, Wow! What a great post. I am so stoked that not only did you find it easy to do but in such a short time. Keep up the great work and sharing your experiences in detail. Cheers, Jay
May 20, 2010 at 08:52 PM
Shaun McCormick:
Great post James. I'm glad to see the docs have helped; if you find any holes or things needing clarification, let us know. As for Quip, I'm in the process of adding moderation to it now. You can see the roadmap here for more info. Hopefully we'll have RC-2 and Quip+moderation out by the time you move over. Thanks again and keep writing. This is really good stuff.
May 21, 2010 at 01:39 AM
devtrench:
Thanks! From first glance it looks like WP comments should move over nicely to Quip since the database tables are pretty similar. I started working on the schema relationships tonight so hopefully I'll have another post about mapping the relationships and migrating that data soon.
May 21, 2010 at 12:25 PM
Shaun McCormick:
Well I hope not *too* soon as I'll be adding a new table, "quipThread", for Quip 0.5 (that said, it will be backwards-compat), as I found out that I needed thread-specific properties rather than just comment-specific. Alas. Whatever you do in Quip 0.4 will work in 0.5, so you can basically ignore this post. :P Great work.
May 21, 2010 at 06:22 AM
Kai Tödter:
James, thanks for this great post! Actually an easy WordPress to MODx Revolution migration would be very useful for me to migrate my site to MODx. Best regards, Kai
May 21, 2010 at 06:49 AM
Gregory Smart:
Very nice post. Can't wait to hear more, I am wanting to get my site migrated too. This will help a lot.
May 21, 2010 at 08:06 AM
Mike Schell:
Nice! xPDO does quite heavily rock. I've recently been playing around with iterating large collections without using tons of memory (this would be a problem if, for example, you were to try to grab and iterate a collection of 2500 WP comments). http://github.com/netProphET/xPDO-Collection-Renderer The solution currently is specific to building up a templated response string from the collection (ie rendering), but it would not be hard at all to modify it for a migration scenario.
May 21, 2010 at 09:58 AM
Jason Coward:
James: Thanks so much for this great contribution. It's long overdue to discuss how xPDO can make importing data from other systems as easy as this. Just a note, you might think about getting small chunks of the WordPress posts at a time rather than all of them. With a large number of posts, this will eat up a large amount of memory. One way to do this would be to explicity sortby('id') and use limit($limit, $offset) to loop over sets of 10 at a time. FWIW, in the future we are going to be adding ways to Iterate large collections like this without the overhead of loading all of the objects in memory and this will solve the problem of getting all the records at once for loops like this.
May 21, 2010 at 10:17 AM
devtrench:
@Jason - I was really impressed at how few lines of code had to be written to do what I did. I'm used to writing straight mysql queries to do this kind of thing and that always ends up taking forever and is really messy. I also pretty much just followed your examples out of the documentation - 80% of the code is just cut and pasted from the forum post you made on generating a schema from a database (which should be in the wiki), and the wiki docs. So thanks for that. As for the memory issues, looks like @netProphET might have a solution for that in the comment above. So far this hasn't been an issue for me - there are about 644 (mostly WP version records) posts and 350+ comments on this blog. So I'm not sure how large a large data set would be. I guess it just depends if I hit the PHP memory limit or not. But a solution for everyone would require low overhead so I'll see if I can build something in. Thanks for your comments!
Aug 10, 2010 at 03:28 AM
i was thinking:
Excellent, Such a great post, very interesting . Thanks for sharing this wonderful information. I was searching like this article.
Sep 30, 2010 at 03:47 AM
Ivan:
Hi, Your article is nice, but i wonder what is ModX? Is that another kind of CMS?
Nov 08, 2010 at 08:40 AM
Asikur Rhaman Reko:
In a single word, i can say that this is a great posting. thanks for such type of good job...