App Engine & 'There are too many files in your application for '

Jeffrey Hawkins (in the comments) mentioned that this has been fixed in the latest version of the SDK. Simply add - ^.*node_modules(/.*)? to your skip_files section. Otherwise the patch is still a option if all else fails.

Most web development workflows today involves using a task running like GruntJS or GulpJS, where in my case I prefer GulpJS.

Along with that problem when developing on App Engine with Python we often have a seperate lib/ folder that stores all our external libraries we might be using.

My usual App Engine app then looks like this:

myapp/  
node_modules/  
lib/  
app.yml

Which is fine and dandy but there is a big problem when you use node here. You'll get the following error from the App Engine local server:

There are too many files in your application for changes in all of them to be monitored. You may have to restart the development server to see some changes to your files.

'There are too many files in your application for '

Well that's not good although expected. The file watcher for the local server can only watch so MANY files.

Great so now what ... ?

Well this is a well-known problem

https://code.google.com/p/googleappengine/issues/detail?id=9300

but has not been address yet, which is a pity.

So let's apply a small band aid to fix this problem while we wait for the issue.

Enough talk, the quick fix !

Find the file watching module

Open up your terminal open the following file, replacing the variables with your own:

nano /Users/{username}/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/mtime_file_watcher.py

This does assume you used the default location for the App Engine installation. If not double check the full error you are getting it will show something like the following:

GAE/1.8.0/google/appengine/tools/devappserver2/mtime_file_watcher.py:82: UserWarning: There are too many files in your application for changes in all of them to be monitored. You may have to restart the development server to see some changes to your files.

'There are too many files in your application for '

In the case of the above error the file we want is:

nano GAE/1.8.0/google/appengine/tools/devappserver2/mtime_file_watcher.py

Add a ignore for folders

In our case we want to ignore node_modules/ and lib/, so in that file on line 105 in the following loop:

for dirname, dirnames, filenames in os.walk

add the following line:

if '/node_modules/' in dirname or '/lib/' in dirname: continue

with all your required folders to ignore. This will simply skip watching any files in those folders. So any changes in those folders will require a restart. Which I'm quite fine with.

Bit of a patch job and will need to be applied on EVERY update till this is fixed but it WORKS !

exclude node_modules from uploading

Another important quick step to save time is to stop the node_modules folder from uploading while deploying. This folder gets quite big which is never even used on live.

To include update your app.yml file under the skip_files section. My section looks like this for example:

skip_files:

- ^(.*/)?.*/assets/.*$
- ^(.*/)?.*/build/.*$
- ^(.*/)?.*/test/.*$
- ^(.*/)?.*/templates/.*$
- ^(.*/)?.*/node_modules/.*$

- ^(assets/.*)
- ^(build/.*)
- ^(test/.*)
- ^(templates/.*)
- ^(node_modules/.*)

- ^(.*/)?#.*#$
- ^(.*/)?.*~$
- ^(.*/)?.*\.DS_Store$
- ^(.*/)?.*\.py[co]$
- ^(.*/)?.*/RCS/.*$
- ^(.*/)?\..*$

Just keeps your remote deployed code clean and ready for action.

Hope this helped you ! Was pulling my hair for quite a while till I got this little bandaid ...

Johann du Toit