App Engine & 'There are too many files in your application for '
Posted on 2015/01/11
Jeffrey Hawkins (in the comments) mentioned that this has been fixed in the latest version of the SDK. Simply add
- ^.*node_modules(/.*)?to your
skip_filessection. Otherwise the patch is still a option if all else fails.
Most web development workflows today involves using a task running like GruntJS or GulpJS, where in my case I prefer GulpJS.
Along with that problem when developing on App Engine with Python we often have a seperate lib/ folder that stores all our external libraries we might be using.
My usual App Engine app then looks like this:
myapp/ node_modules/ lib/ app.yml
Which is fine and dandy but there is a big problem when you use node here. You'll get the following error from the App Engine local server:
There are too many files in your application for changes in all of them to be monitored. You may have to restart the development server to see some changes to your files.
'There are too many files in your application for '
Well that's not good although expected. The file watcher for the local server can only watch so MANY files.
Great so now what ... ?
Well this is a well-known problem
https://code.google.com/p/googleappengine/issues/detail?id=9300but has not been address yet, which is a pity.
So let's apply a small band aid to fix this problem while we wait for the issue.
Enough talk, the quick fix !
Find the file watching module
Open up your terminal open the following file, replacing the variables with your own:
This does assume you used the default location for the App Engine installation. If not double check the full error you are getting it will show something like the following:
GAE/1.8.0/google/appengine/tools/devappserver2/mtime_file_watcher.py:82: UserWarning: There are too many files in your application for changes in all of them to be monitored. You may have to restart the development server to see some changes to your files. 'There are too many files in your application for '
In the case of the above error the file we want is:
Add a ignore for folders
In our case we want to ignore node_modules/ and lib/, so in that file on line 105 in the following loop:
for dirname, dirnames, filenames in os.walk
add the following line:
if '/node_modules/' in dirname or '/lib/' in dirname: continue
with all your required folders to ignore. This will simply skip watching any files in those folders. So any changes in those folders will require a restart. Which I'm quite fine with.
Bit of a patch job and will need to be applied on EVERY update till this is fixed but it WORKS !
exclude node_modules from uploading
Another important quick step to save time is to stop the node_modules folder from uploading while deploying. This folder gets quite big which is never even used on live.
To include update your app.yml file under the skip_files section. My section looks like this for example:
skip_files: - ^(.*/)?.*/assets/.*$ - ^(.*/)?.*/build/.*$ - ^(.*/)?.*/test/.*$ - ^(.*/)?.*/templates/.*$ - ^(.*/)?.*/node_modules/.*$ - ^(assets/.*) - ^(build/.*) - ^(test/.*) - ^(templates/.*) - ^(node_modules/.*) - ^(.*/)?#.*#$ - ^(.*/)?.*~$ - ^(.*/)?.*\.DS_Store$ - ^(.*/)?.*\.py[co]$ - ^(.*/)?.*/RCS/.*$ - ^(.*/)?\..*$
Just keeps your remote deployed code clean and ready for action.
Hope this helped you ! Was pulling my hair for quite a while till I got this little bandaid ...
What's currently keeping me busy
Testing and keeping websites safe
Tech/product of new incubating startups
Advocate and educate on the Google Cloud
Easy prescribed book management
Loadshedding being constantly updated and watched
Secret management for PAAS
National microchip database
Youtube channel of edited meetup talks
Gaming Youtube Channel