Cours : INFO-F422 - Statistical foundations of machine learning - 202425 | UV

  • INFO-F-422 (5 ECTS: Theory : 24h, Exercises : 12h)

    Statistical foundations of machine learning

    Pr. G. Bontempi 
    2024-2025

     

    Course objectives


    Statistical machine learning is the discipline, which aims at extracting knowledge and inferring predictive models from observed data. The course will focus on the statistical notions (like bias, variance, regression, validation), which are necessary to create, identify and assess a predictive model. This course aims to find a good balance between theory and practice by situating most of the theoretical notions in a real context with the help of illustrative case studies (from biology, finance, medicine) and real datasets

    Requirements: courses on statistics and probability and programming skills. 
    Expected prior background: students are supposed to have the following notions and skills. 



    • Annonces Forum
    • Job opportunities, conferences, seminars and hackathons Forum

      This forum is supposed to make available to INFOF422 students news about job opportunities, conferences, seminars, and hackathons in AI, machine learning, data science and analytics. Don't hesitate to contact the professor if you have some event to advertise.

  • UV 2nd session exam 24-25 : important informations to read

    • No project in the 2nd session ! 
    • The final grade (out of 20)  will be obtained as follows:
      • grade   (/20)  = MAX [ 1st sess project  (/10) + UV 2nd sess exam (/10),  UV 2nd sess exam (/20) ]
    • Modalities of the UV theory exam are similar to those of the 1st session.
    • The exam will take place on the morning of August 21st  in the NO building at La Plaine Campus, ULB. You will sit in the NO room mentioned here.
    • If you don't want to do the theory exam, please email me asap and before 6 AM on August 20th: in that case, you will receive the NdP mark. If you don’t inform me and you will not pass the exam, you will receive the mark Absent. In any case, do not come to the exam if you do not intend to pass it: send me an email if you want to contact me.
    • You will need your laptop to pass the UV exam. In case of problems, please get in touch with me as soon as you can. 
    • The test will be online on UV and will last 3h40 (220 minutes). 
    • Schedule:
      • from 7h45:  access to test instructions (it is suggested to copy them somewhere since when the test begins, you won't have access to them anymore)
      • 8h: opening of the test
      • 12h: test closing (whatever the starting time of the individual test)
    • You must be in the assigned exam room by 7:45 at the latest to ensure a smooth start for all of you. Students who arrive later than 8h won’t be accepted for the exam.
    • It is mandatory to bring your laptop, a power cable of at least 3m (or an extension), and writing materials (e.g., paper sheets).
    • The exam room will have easily accessible power plugs for all students and Wifi access. You should check that you can access the eduroam wifi: for more information, look here and  https://support.ulb.be/en/web/support/-/eduroam_win10
    • No allowed telephone usage:
    • During the exam, you will be allowed to consult 
      • some material made accessible from the UV page in a specific section,
      • at most two recto-verso A4 paper  sheets of handwritten notes
    • You can only use the provided Jupyter Notebook to edit your Python code: no other Python IDE is allowed.  Access to programming tools other than Jupyter Notebook is not allowed: the usage of  Google Colab is not allowed
    • You can only use the help() function to access function descriptions. 
    • The following functions will be useful for the exam:
    • During the exam, the only allowed windows on your screen should be the browser to access the UV site, locally installed text editors (no Python IDE), notebook editors (to run the Python code), and the allowed UV documents. All other windows are forbidden. 
    • UV should store your intermediate results every time you change the test page. In any case, it is recommended that you keep all your results (answers + code) on your laptop so that you can re-enter them in case of major issues.
    • If non-authorised applications or windows are opened on your screen, you will be immediately asked to quit the exam, your final mark will be 0/10, and a fraud report will be sent to the Dean.
    • Accessing the UV exam online from a place other than the assigned room in ULB is explicitly forbidden and will be considered a fraudulent attempt.
    • A counter like this one on the right of your screen will show the remaining time. This is the only information you should rely on to know how much time is remaining. When the time is over, the UV system will automatically end the exam. Upload your notebook sufficiently in advance. No exceptions/excuses will be accepted




                •  

                • Introduction to machine learning

                  • Machine learning in the news
                  • Who is a data scientist?
                  • Machine learning and statistics
                  • Deductive and inductive process

                   

                • Foundations of probability and statistics

                  • Foundations of probability and statistics
                  • Interpretation of probability
                  • Conditional probability
                  • Bayes theorem
                  • Random variables
                  • Discrete and continuous r.v.
                  • Multivariate normal distribution.

                   

                • Parametric approaches to estimation

                  • Classical approach .
                  • Estimators.
                  • Bias, variance of an estimator.
                  • Maximum likelihood.
                  • Intervals of confidence.

                   

                • Non parametric approaches to estimation

                  • Computationally intensive methods:
                  • Bootstrap
                  • Combination of estimators.
                  • Regularisation of an estimator.

                   

                • Linear regression modeling:

                  • Simple linear regression
                  • Bias and variance of the estimator
                  • Multiple linear regression.
                  • Least-squares.



                • Supervised learning

                  • Supervised learning: problem definition
                  • Bias/variance decomposition of the generalisation error in regression
                  • Parametric identification.
                  • Structural identification.
                  • Model selection.
                  • Averaging methods

                   

                • Learning algorithms for nonlinear regression

                  • Global vs. divide and conquer models
                  • Feed-forward neural networks
                    • Deep learning
                  • Regression trees
                  • Random forests
                  • Gradient boosting trees
                  • Radial Basis Functions
                  • Local model networks
                  • Local learning

                   

                • Feature selection

                  • Curse of dimensionality
                  • Dimensionality reduction issues
                  • Filters
                    • PCA
                    • Clustering
                    • Information-theory filters
                  • Wrappers
                  • Embedded algorithms
                    • Lasso


                • Supervised classification

                  • Conditional probability
                  • Bayesian classifier
                  • Classification strategies
                  • Algorithms
                    • Linear discriminant
                    • Naive Bayes
                    • nearest-neighbour
                  • Classifier assessment
                    • ROC curve
                    • Specificity, sensitivity, precision
                  • Multiclass classification

                   

                • Handbook

                  G. Bontempi "Statistical foundations of machine learning: the handbook" [link1] 

                  Your comments/remarks/criticisms about the handbook are more than welcome (and could positively affect your marks :-). 

                  To do it, please use the form https://forms.gle/YzPK3zvNp3rXByCQA .

                  A paper version is available at PUB and will be printed on demand. Students who would like to buy it can order it directly at the bookstore's checkout or on the PUB website.


                  The Python code used in the class and mentioned in the handbook is available at

                  https://github.com/gbonte/gbcodepy


                • Bibliography