Importing a datastore backup for local development on App Engine

Been writing a small service over at that crawls and processes data.

Problem is the data is generated and my local internet connection can't get close. So I had to some how import the database from the live datastore into the local instance.

These are the steps I followed to get a local copy of the data imported.

Step 1

Head over to your "datastore admin" screen.

Cheat sheet (just replace with your appid):


Step 2

Next start a backup to a cloud storage bucket on your project.

Not much to say here, just select all the entities you require and hit - "Backup Entities".

Once tasks are done, move on. This might take quite a few minutes and use some storage. Nearline storage would be the perfect choice here, extremely cost effective for your backups.

Step 3

Now pull down the backups to your local machine using gsutil to download all the backup files:

To sync the bucket down use:

gsutil rsync -r gs://{{backup-bucket}}/ ./backups/  

Warning this will pull EVERYTHING in the backup...

Step 4

Right last step, import all the buckets you require into the local App Engine upload_data \  
--url=http://localhost:8080/_ah/remote_api \ 
--application=dev~{{app-id}} \ 
--filename={{folder-of-backups}} \ 


After running your data should now be available locally for your use in development.

Let me know in the comments if I missed anything.

Posted on 2016/02/22 by Johann du Toit

comments powered by Disqus