Show simple item record

dc.contributor.authorPérez, GV
dc.contributor.authorCamargo, CQ
dc.contributor.authorLouis, AA
dc.date.accessioned2021-01-05T13:35:06Z
dc.date.issued2019-05-09
dc.description.abstractDeep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime where classical learning theory would instead predict that they would severely overfit. While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit. In this paper, we provide a new explanation. By applying a very general probability-complexity bound recently derived from algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions. We then provide clear evidence for this strong bias in a model DNN for Boolean functions, as well as in much larger fully conected and convolutional networks trained on CIFAR10 and MNIST. As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems. This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than the more conventional prior over parameter space. If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets. By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood, we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10and for architectures including convolutional and fully connected networks.en_GB
dc.identifier.citationICLR 2019: Seventh International Conference on Learning Representations, 6 - 9 May 2019, New Orleans, Louisiana, USen_GB
dc.identifier.urihttp://hdl.handle.net/10871/124307
dc.language.isoenen_GB
dc.publisherICLRen_GB
dc.relation.urlhttps://iclr.cc/Conferences/2019/Scheduleen_GB
dc.rights© 2019 ICLRen_GB
dc.subjectgeneralizationen_GB
dc.subjectdeep learning theoryen_GB
dc.subjectPAC-Bayesen_GB
dc.subjectGaussian processesen_GB
dc.subjectparameter-function mapen_GB
dc.subjectsimplicity biasen_GB
dc.titleDeep learning generalizes because the parameter-function map is biased towards simple functionsen_GB
dc.typeConference paperen_GB
dc.date.available2021-01-05T13:35:06Z
dc.descriptionThis is the final version. Available from ICLR via the link in this recorden_GB
dc.rights.urihttp://www.rioxx.net/licenses/all-rights-reserveden_GB
rioxxterms.versionVoRen_GB
rioxxterms.licenseref.startdate2019-05-09
rioxxterms.typeConference Paper/Proceeding/Abstracten_GB
refterms.dateFCD2021-01-05T13:31:17Z
refterms.versionFCDVoR
refterms.dateFOA2021-01-05T13:35:12Z
refterms.panelBen_GB


Files in this item

This item appears in the following Collection(s)

Show simple item record